[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 07:58:08.859848 30122 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.106.190:41955,127.29.106.189:39967,127.29.106.188:35115
I20240502 07:58:08.860960 30122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:08.861496 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:08.868726 30128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:08.868849 30127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:08.869402 30130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:08.870196 30122 server_base.cc:1034] running on GCE node
I20240502 07:58:08.870707 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:58:08.870811 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:58:08.870854 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636688870852 us; error 0 us; skew 500 ppm
I20240502 07:58:08.878557 30122 webserver.cc:460] Webserver started at http://127.29.106.190:44021/ using document root <none> and password file <none>
I20240502 07:58:08.879176 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:08.879242 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:08.879467 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:08.881299 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636688849691-30122-0/minicluster-data/master-0-root/instance:
uuid: "fe45358c9b3341f099e7141930e800f4"
format_stamp: "Formatted at 2024-05-02 07:58:08 on dist-test-slave-m6l2"
I20240502 07:58:08.884997 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 07:58:08.887324 30135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:08.888525 30122 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:58:08.888639 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636688849691-30122-0/minicluster-data/master-0-root
uuid: "fe45358c9b3341f099e7141930e800f4"
format_stamp: "Formatted at 2024-05-02 07:58:08 on dist-test-slave-m6l2"
I20240502 07:58:08.888729 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636688849691-30122-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636688849691-30122-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636688849691-30122-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:08.915738 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:08.916649 30122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:08.916847 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:08.924401 30187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.190:41955 every 8 connection(s)
I20240502 07:58:08.924415 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.190:41955
I20240502 07:58:08.925686 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 07:58:08.928025 30188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 07:58:08.928491 30193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:08.929188 30191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:08.929307 30190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:08.930023 30122 server_base.cc:1034] running on GCE node
I20240502 07:58:08.930215 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:58:08.930265 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:58:08.930290 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636688930290 us; error 0 us; skew 500 ppm
I20240502 07:58:08.931479 30122 webserver.cc:460] Webserver started at http://127.29.106.189:43335/ using document root <none> and password file <none>
I20240502 07:58:08.931643 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:08.931705 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:08.931775 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:08.932206 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636688849691-30122-0/minicluster-data/master-1-root/instance:
uuid: "af6faa56b7bb4af081b158f39db0d9bd"
format_stamp: "Formatted at 2024-05-02 07:58:08 on dist-test-slave-m6l2"
I20240502 07:58:08.933720 30188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.190" port: 41955 } has no permanent_uuid. Determining permanent_uuid...
I20240502 07:58:08.934280 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20240502 07:58:08.935387 30198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:08.935715 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:08.935794 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636688849691-30122-0/minicluster-data/master-1-root
uuid: "af6faa56b7bb4af081b158f39db0d9bd"
format_stamp: "Formatted at 2024-05-02 07:58:08 on dist-test-slave-m6l2"
I20240502 07:58:08.935870 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636688849691-30122-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636688849691-30122-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636688849691-30122-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:08.944653 30188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.189" port: 39967 } has no permanent_uuid. Determining permanent_uuid...
W20240502 07:58:08.945453 30139 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.106.189:39967: connect: Connection refused (error 111)
W20240502 07:58:08.946230 30188 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.29.106.189:39967: Network error: Client connection negotiation failed: client connection to 127.29.106.189:39967: connect: Connection refused (error 111)
I20240502 07:58:08.958734 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:08.959206 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:08.964030 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.189:39967
I20240502 07:58:08.964053 30253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.189:39967 every 8 connection(s)
I20240502 07:58:08.964715 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 07:58:08.964956 30254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:08.966315 30254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.190" port: 41955 } has no permanent_uuid. Determining permanent_uuid...
W20240502 07:58:08.966818 30257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:08.967491 30256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:08.968300 30259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:08.968911 30122 server_base.cc:1034] running on GCE node
I20240502 07:58:08.969089 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:58:08.969127 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:58:08.969142 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636688969142 us; error 0 us; skew 500 ppm
I20240502 07:58:08.970173 30122 webserver.cc:460] Webserver started at http://127.29.106.188:40683/ using document root <none> and password file <none>
I20240502 07:58:08.970335 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:08.970402 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:08.970479 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:08.970965 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636688849691-30122-0/minicluster-data/master-2-root/instance:
uuid: "3da78873fde941b2bc323b892c7191e0"
format_stamp: "Formatted at 2024-05-02 07:58:08 on dist-test-slave-m6l2"
I20240502 07:58:08.971426 30254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.189" port: 39967 } has no permanent_uuid. Determining permanent_uuid...
I20240502 07:58:08.972702 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 07:58:08.973786 30266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:08.974071 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:08.974154 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636688849691-30122-0/minicluster-data/master-2-root
uuid: "3da78873fde941b2bc323b892c7191e0"
format_stamp: "Formatted at 2024-05-02 07:58:08 on dist-test-slave-m6l2"
I20240502 07:58:08.974242 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636688849691-30122-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636688849691-30122-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636688849691-30122-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:08.976142 30254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.188" port: 35115 } has no permanent_uuid. Determining permanent_uuid...
W20240502 07:58:08.977417 30254 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.29.106.188:35115: Network error: Client connection negotiation failed: client connection to 127.29.106.188:35115: connect: Connection refused (error 111)
I20240502 07:58:08.995440 30188 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.189" port: 39967 } attempt: 1
I20240502 07:58:08.999763 30188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.188" port: 35115 } has no permanent_uuid. Determining permanent_uuid...
W20240502 07:58:09.001051 30188 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.29.106.188:35115: Network error: Client connection negotiation failed: client connection to 127.29.106.188:35115: connect: Connection refused (error 111)
I20240502 07:58:09.003444 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:09.003820 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:09.008744 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.188:35115
I20240502 07:58:09.008764 30319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.188:35115 every 8 connection(s)
I20240502 07:58:09.009196 30122 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 07:58:09.009667 30320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:09.010759 30320 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.190" port: 41955 } has no permanent_uuid. Determining permanent_uuid...
I20240502 07:58:09.015254 30320 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.189" port: 39967 } has no permanent_uuid. Determining permanent_uuid...
I20240502 07:58:09.019510 30320 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.188" port: 35115 } has no permanent_uuid. Determining permanent_uuid...
I20240502 07:58:09.026079 30320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3da78873fde941b2bc323b892c7191e0: Bootstrap starting.
I20240502 07:58:09.028553 30320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3da78873fde941b2bc323b892c7191e0: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:09.029390 30320 log.cc:824] T 00000000000000000000000000000000 P 3da78873fde941b2bc323b892c7191e0: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:09.029567 30254 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.188" port: 35115 } attempt: 1
I20240502 07:58:09.031493 30320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3da78873fde941b2bc323b892c7191e0: No bootstrap required, opened a new log
I20240502 07:58:09.035008 30254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af6faa56b7bb4af081b158f39db0d9bd: Bootstrap starting.
I20240502 07:58:09.035008 30320 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3da78873fde941b2bc323b892c7191e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe45358c9b3341f099e7141930e800f4" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 41955 } } peers { permanent_uuid: "af6faa56b7bb4af081b158f39db0d9bd" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 39967 } } peers { permanent_uuid: "3da78873fde941b2bc323b892c7191e0" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 35115 } }
I20240502 07:58:09.035339 30320 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3da78873fde941b2bc323b892c7191e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:09.035506 30320 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3da78873fde941b2bc323b892c7191e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da78873fde941b2bc323b892c7191e0, State: Initialized, Role: FOLLOWER
I20240502 07:58:09.035991 30320 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3da78873fde941b2bc323b892c7191e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe45358c9b3341f099e7141930e800f4" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 41955 } } peers { permanent_uuid: "af6faa56b7bb4af081b158f39db0d9bd" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 39967 } } peers { permanent_uuid: "3da78873fde941b2bc323b892c7191e0" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 35115 } }
I20240502 07:58:09.036523 30254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af6faa56b7bb4af081b158f39db0d9bd: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:09.036551 30325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3da78873fde941b2bc323b892c7191e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe45358c9b3341f099e7141930e800f4" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 41955 } } peers { permanent_uuid: "af6faa56b7bb4af081b158f39db0d9bd" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 39967 } } peers { permanent_uuid: "3da78873fde941b2bc323b892c7191e0" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 35115 } } }
I20240502 07:58:09.036669 30325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3da78873fde941b2bc323b892c7191e0 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 07:58:09.037178 30320 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3da78873fde941b2bc323b892c7191e0 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 07:58:09.037699 30254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af6faa56b7bb4af081b158f39db0d9bd: No bootstrap required, opened a new log
I20240502 07:58:09.038218 30254 raft_consensus.cc:348] T 00000000000000000000000000000000 P af6faa56b7bb4af081b158f39db0d9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe45358c9b3341f099e7141930e800f4" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 41955 } } peers { permanent_uuid: "af6faa56b7bb4af081b158f39db0d9bd" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 39967 } } peers { permanent_uuid: "3da78873fde941b2bc323b892c7191e0" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 35115 } }
I20240502 07:58:09.038334 30254 raft_consensus.cc:374] T 00000000000000000000000000000000 P af6faa56b7bb4af081b158f39db0d9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:09.038380 30254 raft_consensus.cc:729] T 00000000000000000000000000000000 P af6faa56b7bb4af081b158f39db0d9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af6faa56b7bb4af081b158f39db0d9bd, State: Initialized, Role: FOLLOWER
I20240502 07:58:09.038513 30254 consensus_queue.cc:260] T 00000000000000000000000000000000 P af6faa56b7bb4af081b158f39db0d9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe45358c9b3341f099e7141930e800f4" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 41955 } } peers { permanent_uuid: "af6faa56b7bb4af081b158f39db0d9bd" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 39967 } } peers { permanent_uuid: "3da78873fde941b2bc323b892c7191e0" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 35115 } }
I20240502 07:58:09.038789 30332 sys_catalog.cc:455] T 00000000000000000000000000000000 P af6faa56b7bb4af081b158f39db0d9bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe45358c9b3341f099e7141930e800f4" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 41955 } } peers { permanent_uuid: "af6faa56b7bb4af081b158f39db0d9bd" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 39967 } } peers { permanent_uuid: "3da78873fde941b2bc323b892c7191e0" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 35115 } } }
I20240502 07:58:09.038897 30332 sys_catalog.cc:458] T 00000000000000000000000000000000 P af6faa56b7bb4af081b158f39db0d9bd [sys.catalog]: This master's current role is: FOLLOWER
I20240502 07:58:09.038991 30254 sys_catalog.cc:564] T 00000000000000000000000000000000 P af6faa56b7bb4af081b158f39db0d9bd [sys.catalog]: configured and running, proceeding with master startup.
W20240502 07:58:09.042376 30349 catalog_manager.cc:1558] T 00000000000000000000000000000000 P af6faa56b7bb4af081b158f39db0d9bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 07:58:09.042376 30338 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3da78873fde941b2bc323b892c7191e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 07:58:09.042472 30349 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 07:58:09.042492 30338 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 07:58:09.044222 30188 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.188" port: 35115 } attempt: 1
I20240502 07:58:09.049351 30188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4: Bootstrap starting.
I20240502 07:58:09.050261 30188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:09.051467 30188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4: No bootstrap required, opened a new log
I20240502 07:58:09.051893 30188 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe45358c9b3341f099e7141930e800f4" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 41955 } } peers { permanent_uuid: "af6faa56b7bb4af081b158f39db0d9bd" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 39967 } } peers { permanent_uuid: "3da78873fde941b2bc323b892c7191e0" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 35115 } }
I20240502 07:58:09.051990 30188 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:09.052013 30188 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe45358c9b3341f099e7141930e800f4, State: Initialized, Role: FOLLOWER
I20240502 07:58:09.052145 30188 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe45358c9b3341f099e7141930e800f4" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 41955 } } peers { permanent_uuid: "af6faa56b7bb4af081b158f39db0d9bd" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 39967 } } peers { permanent_uuid: "3da78873fde941b2bc323b892c7191e0" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 35115 } }
I20240502 07:58:09.052488 30352 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe45358c9b3341f099e7141930e800f4" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 41955 } } peers { permanent_uuid: "af6faa56b7bb4af081b158f39db0d9bd" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 39967 } } peers { permanent_uuid: "3da78873fde941b2bc323b892c7191e0" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 35115 } } }
I20240502 07:58:09.052584 30352 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 07:58:09.052670 30188 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 07:58:09.054596 30122 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240502 07:58:09.054643 30122 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240502 07:58:09.055135 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:09.055245 30363 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 07:58:09.055315 30363 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 07:58:09.057511 30364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:09.058254 30365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:09.061640 30367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:09.062597 30122 server_base.cc:1034] running on GCE node
I20240502 07:58:09.062803 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:58:09.062856 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:58:09.062892 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636689062891 us; error 0 us; skew 500 ppm
I20240502 07:58:09.064160 30122 webserver.cc:460] Webserver started at http://127.29.106.129:45665/ using document root <none> and password file <none>
I20240502 07:58:09.064363 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:09.064421 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:09.064527 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:09.065014 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636688849691-30122-0/minicluster-data/ts-0-root/instance:
uuid: "3b7cbb0b145c46ad8ab5148fee0feed7"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-m6l2"
I20240502 07:58:09.066598 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 07:58:09.067616 30372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.067919 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:09.067997 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636688849691-30122-0/minicluster-data/ts-0-root
uuid: "3b7cbb0b145c46ad8ab5148fee0feed7"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-m6l2"
I20240502 07:58:09.068068 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636688849691-30122-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636688849691-30122-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636688849691-30122-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:09.076349 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:09.076804 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:09.077692 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:09.078632 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:09.078697 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.078781 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:09.078814 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.084990 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.129:33705
I20240502 07:58:09.085062 30435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.129:33705 every 8 connection(s)
I20240502 07:58:09.086122 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:09.088465 30445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:09.089238 30443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:09.091085 30122 server_base.cc:1034] running on GCE node
I20240502 07:58:09.091845 30437 heartbeater.cc:347] Connected to a master server at 127.29.106.190:41955
I20240502 07:58:09.091984 30437 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:09.092381 30436 heartbeater.cc:347] Connected to a master server at 127.29.106.188:35115
I20240502 07:58:09.092468 30436 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:09.092665 30437 heartbeater.cc:511] Master 127.29.106.190:41955 requested a full tablet report, sending...
W20240502 07:58:09.093346 30441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:09.093406 30436 heartbeater.cc:511] Master 127.29.106.188:35115 requested a full tablet report, sending...
I20240502 07:58:09.093673 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:58:09.093734 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:58:09.093730 30352 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:09.093789 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636689093789 us; error 0 us; skew 500 ppm
I20240502 07:58:09.093891 30352 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe45358c9b3341f099e7141930e800f4" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 41955 } } peers { permanent_uuid: "af6faa56b7bb4af081b158f39db0d9bd" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 39967 } } peers { permanent_uuid: "3da78873fde941b2bc323b892c7191e0" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 35115 } }
I20240502 07:58:09.094904 30352 leader_election.cc:288] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af6faa56b7bb4af081b158f39db0d9bd (127.29.106.189:39967), 3da78873fde941b2bc323b892c7191e0 (127.29.106.188:35115)
I20240502 07:58:09.095227 30122 webserver.cc:460] Webserver started at http://127.29.106.130:36391/ using document root <none> and password file <none>
I20240502 07:58:09.095389 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:09.095439 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:09.095521 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:09.096246 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636688849691-30122-0/minicluster-data/ts-1-root/instance:
uuid: "67e75196581a47cfa5befc23e8bce1ee"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-m6l2"
I20240502 07:58:09.096478 30152 ts_manager.cc:194] Registered new tserver with Master: 3b7cbb0b145c46ad8ab5148fee0feed7 (127.29.106.129:33705)
I20240502 07:58:09.096879 30284 ts_manager.cc:194] Registered new tserver with Master: 3b7cbb0b145c46ad8ab5148fee0feed7 (127.29.106.129:33705)
I20240502 07:58:09.097460 30228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fe45358c9b3341f099e7141930e800f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af6faa56b7bb4af081b158f39db0d9bd" is_pre_election: true
I20240502 07:58:09.097656 30228 raft_consensus.cc:2454] T 00000000000000000000000000000000 P af6faa56b7bb4af081b158f39db0d9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe45358c9b3341f099e7141930e800f4 in term 0.
I20240502 07:58:09.095440 30294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fe45358c9b3341f099e7141930e800f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da78873fde941b2bc323b892c7191e0" is_pre_election: true
I20240502 07:58:09.098021 30139 leader_election.cc:302] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af6faa56b7bb4af081b158f39db0d9bd, fe45358c9b3341f099e7141930e800f4; no voters:
I20240502 07:58:09.098119 30294 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3da78873fde941b2bc323b892c7191e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe45358c9b3341f099e7141930e800f4 in term 0.
I20240502 07:58:09.098377 30438 heartbeater.cc:347] Connected to a master server at 127.29.106.189:39967
I20240502 07:58:09.098405 30352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:09.098443 30438 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:09.098464 30352 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:09.098506 30352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:09.098655 30438 heartbeater.cc:511] Master 127.29.106.189:39967 requested a full tablet report, sending...
I20240502 07:58:09.099017 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240502 07:58:09.099335 30218 ts_manager.cc:194] Registered new tserver with Master: 3b7cbb0b145c46ad8ab5148fee0feed7 (127.29.106.129:33705)
I20240502 07:58:09.099509 30352 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe45358c9b3341f099e7141930e800f4" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 41955 } } peers { permanent_uuid: "af6faa56b7bb4af081b158f39db0d9bd" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 39967 } } peers { permanent_uuid: "3da78873fde941b2bc323b892c7191e0" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 35115 } }
I20240502 07:58:09.099850 30352 leader_election.cc:288] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4 [CANDIDATE]: Term 1 election: Requested vote from peers af6faa56b7bb4af081b158f39db0d9bd (127.29.106.189:39967), 3da78873fde941b2bc323b892c7191e0 (127.29.106.188:35115)
I20240502 07:58:09.100342 30294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fe45358c9b3341f099e7141930e800f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da78873fde941b2bc323b892c7191e0"
I20240502 07:58:09.100327 30228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fe45358c9b3341f099e7141930e800f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af6faa56b7bb4af081b158f39db0d9bd"
I20240502 07:58:09.100490 30294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3da78873fde941b2bc323b892c7191e0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:09.100512 30228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af6faa56b7bb4af081b158f39db0d9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:09.100358 30451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.100831 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:09.100896 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636688849691-30122-0/minicluster-data/ts-1-root
uuid: "67e75196581a47cfa5befc23e8bce1ee"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-m6l2"
I20240502 07:58:09.100957 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636688849691-30122-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636688849691-30122-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636688849691-30122-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:09.101434 30294 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3da78873fde941b2bc323b892c7191e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe45358c9b3341f099e7141930e800f4 in term 1.
I20240502 07:58:09.101505 30228 raft_consensus.cc:2454] T 00000000000000000000000000000000 P af6faa56b7bb4af081b158f39db0d9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe45358c9b3341f099e7141930e800f4 in term 1.
I20240502 07:58:09.101713 30136 leader_election.cc:302] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da78873fde941b2bc323b892c7191e0, fe45358c9b3341f099e7141930e800f4; no voters:
I20240502 07:58:09.101898 30352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:09.102115 30352 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4 [term 1 LEADER]: Becoming Leader. State: Replica: fe45358c9b3341f099e7141930e800f4, State: Running, Role: LEADER
I20240502 07:58:09.102409 30352 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe45358c9b3341f099e7141930e800f4" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 41955 } } peers { permanent_uuid: "af6faa56b7bb4af081b158f39db0d9bd" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 39967 } } peers { permanent_uuid: "3da78873fde941b2bc323b892c7191e0" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 35115 } }
I20240502 07:58:09.104377 30456 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe45358c9b3341f099e7141930e800f4. Latest consensus state: current_term: 1 leader_uuid: "fe45358c9b3341f099e7141930e800f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe45358c9b3341f099e7141930e800f4" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 41955 } } peers { permanent_uuid: "af6faa56b7bb4af081b158f39db0d9bd" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 39967 } } peers { permanent_uuid: "3da78873fde941b2bc323b892c7191e0" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 35115 } } }
I20240502 07:58:09.104483 30456 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4 [sys.catalog]: This master's current role is: LEADER
I20240502 07:58:09.105233 30458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 07:58:09.106164 30458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 07:58:09.108681 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:09.109057 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:09.109319 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:09.109802 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:09.109844 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.109887 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:09.109926 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.111819 30294 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3da78873fde941b2bc323b892c7191e0 [term 1 FOLLOWER]: Refusing update from remote peer fe45358c9b3341f099e7141930e800f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:09.112571 30352 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da78873fde941b2bc323b892c7191e0" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 35115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:09.112571 30228 raft_consensus.cc:1261] T 00000000000000000000000000000000 P af6faa56b7bb4af081b158f39db0d9bd [term 1 FOLLOWER]: Refusing update from remote peer fe45358c9b3341f099e7141930e800f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:09.113438 30352 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af6faa56b7bb4af081b158f39db0d9bd" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 39967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:09.114574 30325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3da78873fde941b2bc323b892c7191e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe45358c9b3341f099e7141930e800f4. Latest consensus state: current_term: 1 leader_uuid: "fe45358c9b3341f099e7141930e800f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe45358c9b3341f099e7141930e800f4" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 41955 } } peers { permanent_uuid: "af6faa56b7bb4af081b158f39db0d9bd" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 39967 } } peers { permanent_uuid: "3da78873fde941b2bc323b892c7191e0" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 35115 } } }
I20240502 07:58:09.114704 30325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3da78873fde941b2bc323b892c7191e0 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 07:58:09.114676 30332 sys_catalog.cc:455] T 00000000000000000000000000000000 P af6faa56b7bb4af081b158f39db0d9bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe45358c9b3341f099e7141930e800f4. Latest consensus state: current_term: 1 leader_uuid: "fe45358c9b3341f099e7141930e800f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe45358c9b3341f099e7141930e800f4" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 41955 } } peers { permanent_uuid: "af6faa56b7bb4af081b158f39db0d9bd" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 39967 } } peers { permanent_uuid: "3da78873fde941b2bc323b892c7191e0" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 35115 } } }
I20240502 07:58:09.114770 30332 sys_catalog.cc:458] T 00000000000000000000000000000000 P af6faa56b7bb4af081b158f39db0d9bd [sys.catalog]: This master's current role is: FOLLOWER
I20240502 07:58:09.116034 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.130:37477
I20240502 07:58:09.116420 30456 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fe45358c9b3341f099e7141930e800f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe45358c9b3341f099e7141930e800f4" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 41955 } } peers { permanent_uuid: "af6faa56b7bb4af081b158f39db0d9bd" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 39967 } } peers { permanent_uuid: "3da78873fde941b2bc323b892c7191e0" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 35115 } } }
I20240502 07:58:09.116515 30456 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4 [sys.catalog]: This master's current role is: LEADER
I20240502 07:58:09.116698 30325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3da78873fde941b2bc323b892c7191e0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fe45358c9b3341f099e7141930e800f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe45358c9b3341f099e7141930e800f4" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 41955 } } peers { permanent_uuid: "af6faa56b7bb4af081b158f39db0d9bd" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 39967 } } peers { permanent_uuid: "3da78873fde941b2bc323b892c7191e0" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 35115 } } }
I20240502 07:58:09.116786 30325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3da78873fde941b2bc323b892c7191e0 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 07:58:09.117205 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 07:58:09.117314 30459 mvcc.cc:205] Tried to move back new op lower bound from 7023151878595223552 to 7023151878566297600. Current Snapshot: MvccSnapshot[applied={T|T < 7023151878595223552}]
I20240502 07:58:09.118722 30522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.130:37477 every 8 connection(s)
I20240502 07:58:09.119851 30456 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fe45358c9b3341f099e7141930e800f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe45358c9b3341f099e7141930e800f4" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 41955 } } peers { permanent_uuid: "af6faa56b7bb4af081b158f39db0d9bd" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 39967 } } peers { permanent_uuid: "3da78873fde941b2bc323b892c7191e0" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 35115 } } }
I20240502 07:58:09.119972 30456 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4 [sys.catalog]: This master's current role is: LEADER
W20240502 07:58:09.120630 30529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:09.121027 30532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:09.121033 30332 sys_catalog.cc:455] T 00000000000000000000000000000000 P af6faa56b7bb4af081b158f39db0d9bd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fe45358c9b3341f099e7141930e800f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe45358c9b3341f099e7141930e800f4" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 41955 } } peers { permanent_uuid: "af6faa56b7bb4af081b158f39db0d9bd" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 39967 } } peers { permanent_uuid: "3da78873fde941b2bc323b892c7191e0" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 35115 } } }
I20240502 07:58:09.121423 30332 sys_catalog.cc:458] T 00000000000000000000000000000000 P af6faa56b7bb4af081b158f39db0d9bd [sys.catalog]: This master's current role is: FOLLOWER
I20240502 07:58:09.121878 30458 catalog_manager.cc:1347] Generated new cluster ID: 06e61be4c92d43e08f32f53b570c9993
I20240502 07:58:09.121973 30458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 07:58:09.122228 30122 server_base.cc:1034] running on GCE node
W20240502 07:58:09.122416 30530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:09.122617 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:58:09.122654 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:58:09.122676 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636689122676 us; error 0 us; skew 500 ppm
I20240502 07:58:09.123848 30122 webserver.cc:460] Webserver started at http://127.29.106.131:38809/ using document root <none> and password file <none>
I20240502 07:58:09.124001 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:09.124048 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:09.124114 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:09.124575 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636688849691-30122-0/minicluster-data/ts-2-root/instance:
uuid: "6117e90930294453ada6ea1fcca62663"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-m6l2"
I20240502 07:58:09.127938 30526 heartbeater.cc:347] Connected to a master server at 127.29.106.190:41955
I20240502 07:58:09.128022 30526 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:09.129048 30526 heartbeater.cc:511] Master 127.29.106.190:41955 requested a full tablet report, sending...
I20240502 07:58:09.129616 30152 ts_manager.cc:194] Registered new tserver with Master: 67e75196581a47cfa5befc23e8bce1ee (127.29.106.130:37477)
I20240502 07:58:09.131004 30527 heartbeater.cc:347] Connected to a master server at 127.29.106.189:39967
I20240502 07:58:09.131094 30527 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:09.131294 30527 heartbeater.cc:511] Master 127.29.106.189:39967 requested a full tablet report, sending...
I20240502 07:58:09.131839 30218 ts_manager.cc:194] Registered new tserver with Master: 67e75196581a47cfa5befc23e8bce1ee (127.29.106.130:37477)
I20240502 07:58:09.131927 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.000s
I20240502 07:58:09.132787 30524 heartbeater.cc:347] Connected to a master server at 127.29.106.188:35115
I20240502 07:58:09.132879 30524 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:09.133042 30541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.133059 30524 heartbeater.cc:511] Master 127.29.106.188:35115 requested a full tablet report, sending...
I20240502 07:58:09.133268 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:09.133332 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636688849691-30122-0/minicluster-data/ts-2-root
uuid: "6117e90930294453ada6ea1fcca62663"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-m6l2"
I20240502 07:58:09.133397 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636688849691-30122-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636688849691-30122-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636688849691-30122-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:09.133733 30284 ts_manager.cc:194] Registered new tserver with Master: 67e75196581a47cfa5befc23e8bce1ee (127.29.106.130:37477)
I20240502 07:58:09.152623 30458 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 07:58:09.154062 30458 catalog_manager.cc:1504] Loading token signing keys...
I20240502 07:58:09.164012 30458 catalog_manager.cc:5857] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4: Generated new TSK 0
I20240502 07:58:09.164880 30458 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 07:58:09.182765 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:09.183231 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:09.183579 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:09.184144 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:09.184194 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.184237 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:09.184295 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.190124 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.131:34645
I20240502 07:58:09.190192 30605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.131:34645 every 8 connection(s)
I20240502 07:58:09.195941 30607 heartbeater.cc:347] Connected to a master server at 127.29.106.190:41955
I20240502 07:58:09.196055 30607 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:09.196305 30607 heartbeater.cc:511] Master 127.29.106.190:41955 requested a full tablet report, sending...
I20240502 07:58:09.196914 30152 ts_manager.cc:194] Registered new tserver with Master: 6117e90930294453ada6ea1fcca62663 (127.29.106.131:34645)
I20240502 07:58:09.197610 30608 heartbeater.cc:347] Connected to a master server at 127.29.106.189:39967
I20240502 07:58:09.197695 30608 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:09.197885 30608 heartbeater.cc:511] Master 127.29.106.189:39967 requested a full tablet report, sending...
I20240502 07:58:09.198120 30606 heartbeater.cc:347] Connected to a master server at 127.29.106.188:35115
I20240502 07:58:09.198195 30606 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:09.198246 30152 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51094
I20240502 07:58:09.198381 30606 heartbeater.cc:511] Master 127.29.106.188:35115 requested a full tablet report, sending...
I20240502 07:58:09.198904 30284 ts_manager.cc:194] Registered new tserver with Master: 6117e90930294453ada6ea1fcca62663 (127.29.106.131:34645)
I20240502 07:58:09.198904 30218 ts_manager.cc:194] Registered new tserver with Master: 6117e90930294453ada6ea1fcca62663 (127.29.106.131:34645)
I20240502 07:58:09.199321 30122 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.00859796s
I20240502 07:58:10.043573 30349 catalog_manager.cc:1259] Loaded cluster ID: 06e61be4c92d43e08f32f53b570c9993
I20240502 07:58:10.043676 30349 catalog_manager.cc:1552] T 00000000000000000000000000000000 P af6faa56b7bb4af081b158f39db0d9bd: loading cluster ID for follower catalog manager: success
I20240502 07:58:10.043573 30338 catalog_manager.cc:1259] Loaded cluster ID: 06e61be4c92d43e08f32f53b570c9993
I20240502 07:58:10.043777 30338 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3da78873fde941b2bc323b892c7191e0: loading cluster ID for follower catalog manager: success
I20240502 07:58:10.045046 30349 catalog_manager.cc:1574] T 00000000000000000000000000000000 P af6faa56b7bb4af081b158f39db0d9bd: acquiring CA information for follower catalog manager: success
I20240502 07:58:10.045064 30338 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3da78873fde941b2bc323b892c7191e0: acquiring CA information for follower catalog manager: success
I20240502 07:58:10.045749 30338 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3da78873fde941b2bc323b892c7191e0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 07:58:10.045749 30349 catalog_manager.cc:1602] T 00000000000000000000000000000000 P af6faa56b7bb4af081b158f39db0d9bd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 07:58:10.100466 30152 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51070
I20240502 07:58:10.132134 30152 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51080
I20240502 07:58:10.199733 30607 heartbeater.cc:503] Master 127.29.106.190:41955 was elected leader, sending a full tablet report...
I20240502 07:58:11.054967 30362 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:58:11.101570 30437 heartbeater.cc:503] Master 127.29.106.190:41955 was elected leader, sending a full tablet report...
I20240502 07:58:11.133522 30526 heartbeater.cc:503] Master 127.29.106.190:41955 was elected leader, sending a full tablet report...
I20240502 07:58:11.252110 30122 test_util.cc:268] Using random seed: -152657016
I20240502 07:58:11.267877 30152 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51102:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 07:58:11.291785 30483 tablet_service.cc:1467] Processing CreateTablet for tablet 7851822f3fbc449e9c463dadeaf3f92d (DEFAULT_TABLE table=test-workload [id=624782e805b54d579249bd79caa2790f]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 07:58:11.292256 30400 tablet_service.cc:1467] Processing CreateTablet for tablet c05ae203ae1f463590579bfe11bf7932 (DEFAULT_TABLE table=test-workload [id=624782e805b54d579249bd79caa2790f]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 07:58:11.292415 30483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7851822f3fbc449e9c463dadeaf3f92d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:11.292680 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c05ae203ae1f463590579bfe11bf7932. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:11.291785 30570 tablet_service.cc:1467] Processing CreateTablet for tablet d332a5e31e21465499833f960bf57e9b (DEFAULT_TABLE table=test-workload [id=624782e805b54d579249bd79caa2790f]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 07:58:11.295435 30570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d332a5e31e21465499833f960bf57e9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:11.295532 30642 tablet_bootstrap.cc:492] T 7851822f3fbc449e9c463dadeaf3f92d P 67e75196581a47cfa5befc23e8bce1ee: Bootstrap starting.
I20240502 07:58:11.292243 30399 tablet_service.cc:1467] Processing CreateTablet for tablet 08bdbe000f484a47a3cf825487fa498d (DEFAULT_TABLE table=test-workload [id=624782e805b54d579249bd79caa2790f]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 07:58:11.295985 30399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08bdbe000f484a47a3cf825487fa498d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:11.296586 30643 tablet_bootstrap.cc:492] T c05ae203ae1f463590579bfe11bf7932 P 3b7cbb0b145c46ad8ab5148fee0feed7: Bootstrap starting.
I20240502 07:58:11.296675 30642 tablet_bootstrap.cc:654] T 7851822f3fbc449e9c463dadeaf3f92d P 67e75196581a47cfa5befc23e8bce1ee: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:11.297737 30643 tablet_bootstrap.cc:654] T c05ae203ae1f463590579bfe11bf7932 P 3b7cbb0b145c46ad8ab5148fee0feed7: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:11.297952 30642 tablet_bootstrap.cc:492] T 7851822f3fbc449e9c463dadeaf3f92d P 67e75196581a47cfa5befc23e8bce1ee: No bootstrap required, opened a new log
I20240502 07:58:11.298079 30642 ts_tablet_manager.cc:1367] T 7851822f3fbc449e9c463dadeaf3f92d P 67e75196581a47cfa5befc23e8bce1ee: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:11.298591 30642 raft_consensus.cc:348] T 7851822f3fbc449e9c463dadeaf3f92d P 67e75196581a47cfa5befc23e8bce1ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67e75196581a47cfa5befc23e8bce1ee" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 37477 } }
I20240502 07:58:11.298722 30642 raft_consensus.cc:374] T 7851822f3fbc449e9c463dadeaf3f92d P 67e75196581a47cfa5befc23e8bce1ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:11.298753 30642 raft_consensus.cc:729] T 7851822f3fbc449e9c463dadeaf3f92d P 67e75196581a47cfa5befc23e8bce1ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e75196581a47cfa5befc23e8bce1ee, State: Initialized, Role: FOLLOWER
I20240502 07:58:11.298908 30642 consensus_queue.cc:260] T 7851822f3fbc449e9c463dadeaf3f92d P 67e75196581a47cfa5befc23e8bce1ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67e75196581a47cfa5befc23e8bce1ee" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 37477 } }
I20240502 07:58:11.299000 30642 raft_consensus.cc:388] T 7851822f3fbc449e9c463dadeaf3f92d P 67e75196581a47cfa5befc23e8bce1ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:58:11.299057 30642 raft_consensus.cc:482] T 7851822f3fbc449e9c463dadeaf3f92d P 67e75196581a47cfa5befc23e8bce1ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:58:11.299103 30642 raft_consensus.cc:3037] T 7851822f3fbc449e9c463dadeaf3f92d P 67e75196581a47cfa5befc23e8bce1ee [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:11.299190 30645 tablet_bootstrap.cc:492] T d332a5e31e21465499833f960bf57e9b P 6117e90930294453ada6ea1fcca62663: Bootstrap starting.
I20240502 07:58:11.300086 30642 raft_consensus.cc:504] T 7851822f3fbc449e9c463dadeaf3f92d P 67e75196581a47cfa5befc23e8bce1ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67e75196581a47cfa5befc23e8bce1ee" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 37477 } }
I20240502 07:58:11.300220 30642 leader_election.cc:302] T 7851822f3fbc449e9c463dadeaf3f92d P 67e75196581a47cfa5befc23e8bce1ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67e75196581a47cfa5befc23e8bce1ee; no voters:
I20240502 07:58:11.300451 30642 leader_election.cc:288] T 7851822f3fbc449e9c463dadeaf3f92d P 67e75196581a47cfa5befc23e8bce1ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:58:11.300515 30643 tablet_bootstrap.cc:492] T c05ae203ae1f463590579bfe11bf7932 P 3b7cbb0b145c46ad8ab5148fee0feed7: No bootstrap required, opened a new log
I20240502 07:58:11.300642 30643 ts_tablet_manager.cc:1367] T c05ae203ae1f463590579bfe11bf7932 P 3b7cbb0b145c46ad8ab5148fee0feed7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 07:58:11.300998 30642 ts_tablet_manager.cc:1398] T 7851822f3fbc449e9c463dadeaf3f92d P 67e75196581a47cfa5befc23e8bce1ee: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 07:58:11.301153 30643 raft_consensus.cc:348] T c05ae203ae1f463590579bfe11bf7932 P 3b7cbb0b145c46ad8ab5148fee0feed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7cbb0b145c46ad8ab5148fee0feed7" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33705 } }
I20240502 07:58:11.301313 30643 raft_consensus.cc:374] T c05ae203ae1f463590579bfe11bf7932 P 3b7cbb0b145c46ad8ab5148fee0feed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:11.301373 30643 raft_consensus.cc:729] T c05ae203ae1f463590579bfe11bf7932 P 3b7cbb0b145c46ad8ab5148fee0feed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b7cbb0b145c46ad8ab5148fee0feed7, State: Initialized, Role: FOLLOWER
I20240502 07:58:11.301400 30645 tablet_bootstrap.cc:654] T d332a5e31e21465499833f960bf57e9b P 6117e90930294453ada6ea1fcca62663: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:11.301523 30643 consensus_queue.cc:260] T c05ae203ae1f463590579bfe11bf7932 P 3b7cbb0b145c46ad8ab5148fee0feed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7cbb0b145c46ad8ab5148fee0feed7" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33705 } }
I20240502 07:58:11.301616 30643 raft_consensus.cc:388] T c05ae203ae1f463590579bfe11bf7932 P 3b7cbb0b145c46ad8ab5148fee0feed7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:58:11.301677 30643 raft_consensus.cc:482] T c05ae203ae1f463590579bfe11bf7932 P 3b7cbb0b145c46ad8ab5148fee0feed7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:58:11.301733 30643 raft_consensus.cc:3037] T c05ae203ae1f463590579bfe11bf7932 P 3b7cbb0b145c46ad8ab5148fee0feed7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:11.302961 30645 tablet_bootstrap.cc:492] T d332a5e31e21465499833f960bf57e9b P 6117e90930294453ada6ea1fcca62663: No bootstrap required, opened a new log
I20240502 07:58:11.303231 30643 raft_consensus.cc:504] T c05ae203ae1f463590579bfe11bf7932 P 3b7cbb0b145c46ad8ab5148fee0feed7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7cbb0b145c46ad8ab5148fee0feed7" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33705 } }
I20240502 07:58:11.303267 30648 raft_consensus.cc:2781] T 7851822f3fbc449e9c463dadeaf3f92d P 67e75196581a47cfa5befc23e8bce1ee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:11.303397 30643 leader_election.cc:302] T c05ae203ae1f463590579bfe11bf7932 P 3b7cbb0b145c46ad8ab5148fee0feed7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b7cbb0b145c46ad8ab5148fee0feed7; no voters:
I20240502 07:58:11.303450 30648 raft_consensus.cc:686] T 7851822f3fbc449e9c463dadeaf3f92d P 67e75196581a47cfa5befc23e8bce1ee [term 1 LEADER]: Becoming Leader. State: Replica: 67e75196581a47cfa5befc23e8bce1ee, State: Running, Role: LEADER
I20240502 07:58:11.303586 30648 consensus_queue.cc:237] T 7851822f3fbc449e9c463dadeaf3f92d P 67e75196581a47cfa5befc23e8bce1ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67e75196581a47cfa5befc23e8bce1ee" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 37477 } }
I20240502 07:58:11.303656 30643 leader_election.cc:288] T c05ae203ae1f463590579bfe11bf7932 P 3b7cbb0b145c46ad8ab5148fee0feed7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:58:11.304044 30643 ts_tablet_manager.cc:1398] T c05ae203ae1f463590579bfe11bf7932 P 3b7cbb0b145c46ad8ab5148fee0feed7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 07:58:11.304342 30643 tablet_bootstrap.cc:492] T 08bdbe000f484a47a3cf825487fa498d P 3b7cbb0b145c46ad8ab5148fee0feed7: Bootstrap starting.
I20240502 07:58:11.305440 30643 tablet_bootstrap.cc:654] T 08bdbe000f484a47a3cf825487fa498d P 3b7cbb0b145c46ad8ab5148fee0feed7: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:11.307750 30645 ts_tablet_manager.cc:1367] T d332a5e31e21465499833f960bf57e9b P 6117e90930294453ada6ea1fcca62663: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240502 07:58:11.308432 30650 raft_consensus.cc:2781] T c05ae203ae1f463590579bfe11bf7932 P 3b7cbb0b145c46ad8ab5148fee0feed7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:11.308331 30645 raft_consensus.cc:348] T d332a5e31e21465499833f960bf57e9b P 6117e90930294453ada6ea1fcca62663 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6117e90930294453ada6ea1fcca62663" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34645 } }
I20240502 07:58:11.308523 30645 raft_consensus.cc:374] T d332a5e31e21465499833f960bf57e9b P 6117e90930294453ada6ea1fcca62663 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:11.308566 30645 raft_consensus.cc:729] T d332a5e31e21465499833f960bf57e9b P 6117e90930294453ada6ea1fcca62663 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6117e90930294453ada6ea1fcca62663, State: Initialized, Role: FOLLOWER
I20240502 07:58:11.308307 30152 catalog_manager.cc:5484] T 7851822f3fbc449e9c463dadeaf3f92d P 67e75196581a47cfa5befc23e8bce1ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 67e75196581a47cfa5befc23e8bce1ee (127.29.106.130). New cstate: current_term: 1 leader_uuid: "67e75196581a47cfa5befc23e8bce1ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67e75196581a47cfa5befc23e8bce1ee" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 37477 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:11.308658 30650 raft_consensus.cc:686] T c05ae203ae1f463590579bfe11bf7932 P 3b7cbb0b145c46ad8ab5148fee0feed7 [term 1 LEADER]: Becoming Leader. State: Replica: 3b7cbb0b145c46ad8ab5148fee0feed7, State: Running, Role: LEADER
I20240502 07:58:11.308745 30645 consensus_queue.cc:260] T d332a5e31e21465499833f960bf57e9b P 6117e90930294453ada6ea1fcca62663 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6117e90930294453ada6ea1fcca62663" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34645 } }
I20240502 07:58:11.308791 30650 consensus_queue.cc:237] T c05ae203ae1f463590579bfe11bf7932 P 3b7cbb0b145c46ad8ab5148fee0feed7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7cbb0b145c46ad8ab5148fee0feed7" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33705 } }
I20240502 07:58:11.308845 30645 raft_consensus.cc:388] T d332a5e31e21465499833f960bf57e9b P 6117e90930294453ada6ea1fcca62663 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:58:11.308892 30645 raft_consensus.cc:482] T d332a5e31e21465499833f960bf57e9b P 6117e90930294453ada6ea1fcca62663 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:58:11.308946 30645 raft_consensus.cc:3037] T d332a5e31e21465499833f960bf57e9b P 6117e90930294453ada6ea1fcca62663 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:11.309799 30643 tablet_bootstrap.cc:492] T 08bdbe000f484a47a3cf825487fa498d P 3b7cbb0b145c46ad8ab5148fee0feed7: No bootstrap required, opened a new log
I20240502 07:58:11.309965 30643 ts_tablet_manager.cc:1367] T 08bdbe000f484a47a3cf825487fa498d P 3b7cbb0b145c46ad8ab5148fee0feed7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 07:58:11.310037 30645 raft_consensus.cc:504] T d332a5e31e21465499833f960bf57e9b P 6117e90930294453ada6ea1fcca62663 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6117e90930294453ada6ea1fcca62663" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34645 } }
I20240502 07:58:11.310175 30645 leader_election.cc:302] T d332a5e31e21465499833f960bf57e9b P 6117e90930294453ada6ea1fcca62663 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6117e90930294453ada6ea1fcca62663; no voters:
I20240502 07:58:11.310388 30645 leader_election.cc:288] T d332a5e31e21465499833f960bf57e9b P 6117e90930294453ada6ea1fcca62663 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:58:11.310678 30643 raft_consensus.cc:348] T 08bdbe000f484a47a3cf825487fa498d P 3b7cbb0b145c46ad8ab5148fee0feed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7cbb0b145c46ad8ab5148fee0feed7" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33705 } }
I20240502 07:58:11.310729 30645 ts_tablet_manager.cc:1398] T d332a5e31e21465499833f960bf57e9b P 6117e90930294453ada6ea1fcca62663: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 07:58:11.310840 30643 raft_consensus.cc:374] T 08bdbe000f484a47a3cf825487fa498d P 3b7cbb0b145c46ad8ab5148fee0feed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:11.310961 30643 raft_consensus.cc:729] T 08bdbe000f484a47a3cf825487fa498d P 3b7cbb0b145c46ad8ab5148fee0feed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b7cbb0b145c46ad8ab5148fee0feed7, State: Initialized, Role: FOLLOWER
I20240502 07:58:11.311158 30643 consensus_queue.cc:260] T 08bdbe000f484a47a3cf825487fa498d P 3b7cbb0b145c46ad8ab5148fee0feed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7cbb0b145c46ad8ab5148fee0feed7" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33705 } }
I20240502 07:58:11.311319 30643 raft_consensus.cc:388] T 08bdbe000f484a47a3cf825487fa498d P 3b7cbb0b145c46ad8ab5148fee0feed7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:58:11.311403 30643 raft_consensus.cc:482] T 08bdbe000f484a47a3cf825487fa498d P 3b7cbb0b145c46ad8ab5148fee0feed7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:58:11.311497 30643 raft_consensus.cc:3037] T 08bdbe000f484a47a3cf825487fa498d P 3b7cbb0b145c46ad8ab5148fee0feed7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:11.312597 30643 raft_consensus.cc:504] T 08bdbe000f484a47a3cf825487fa498d P 3b7cbb0b145c46ad8ab5148fee0feed7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7cbb0b145c46ad8ab5148fee0feed7" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33705 } }
I20240502 07:58:11.312727 30643 leader_election.cc:302] T 08bdbe000f484a47a3cf825487fa498d P 3b7cbb0b145c46ad8ab5148fee0feed7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b7cbb0b145c46ad8ab5148fee0feed7; no voters:
I20240502 07:58:11.312811 30643 leader_election.cc:288] T 08bdbe000f484a47a3cf825487fa498d P 3b7cbb0b145c46ad8ab5148fee0feed7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:58:11.313068 30659 raft_consensus.cc:2781] T 08bdbe000f484a47a3cf825487fa498d P 3b7cbb0b145c46ad8ab5148fee0feed7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:11.313119 30643 ts_tablet_manager.cc:1398] T 08bdbe000f484a47a3cf825487fa498d P 3b7cbb0b145c46ad8ab5148fee0feed7: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 07:58:11.313313 30659 raft_consensus.cc:686] T 08bdbe000f484a47a3cf825487fa498d P 3b7cbb0b145c46ad8ab5148fee0feed7 [term 1 LEADER]: Becoming Leader. State: Replica: 3b7cbb0b145c46ad8ab5148fee0feed7, State: Running, Role: LEADER
I20240502 07:58:11.313457 30659 consensus_queue.cc:237] T 08bdbe000f484a47a3cf825487fa498d P 3b7cbb0b145c46ad8ab5148fee0feed7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7cbb0b145c46ad8ab5148fee0feed7" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33705 } }
I20240502 07:58:11.314724 30150 catalog_manager.cc:5484] T c05ae203ae1f463590579bfe11bf7932 P 3b7cbb0b145c46ad8ab5148fee0feed7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b7cbb0b145c46ad8ab5148fee0feed7 (127.29.106.129). New cstate: current_term: 1 leader_uuid: "3b7cbb0b145c46ad8ab5148fee0feed7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7cbb0b145c46ad8ab5148fee0feed7" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33705 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:11.315023 30150 catalog_manager.cc:5484] T 08bdbe000f484a47a3cf825487fa498d P 3b7cbb0b145c46ad8ab5148fee0feed7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b7cbb0b145c46ad8ab5148fee0feed7 (127.29.106.129). New cstate: current_term: 1 leader_uuid: "3b7cbb0b145c46ad8ab5148fee0feed7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7cbb0b145c46ad8ab5148fee0feed7" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33705 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:11.317909 30657 raft_consensus.cc:2781] T d332a5e31e21465499833f960bf57e9b P 6117e90930294453ada6ea1fcca62663 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:11.318120 30657 raft_consensus.cc:686] T d332a5e31e21465499833f960bf57e9b P 6117e90930294453ada6ea1fcca62663 [term 1 LEADER]: Becoming Leader. State: Replica: 6117e90930294453ada6ea1fcca62663, State: Running, Role: LEADER
I20240502 07:58:11.318259 30657 consensus_queue.cc:237] T d332a5e31e21465499833f960bf57e9b P 6117e90930294453ada6ea1fcca62663 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6117e90930294453ada6ea1fcca62663" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34645 } }
I20240502 07:58:11.319823 30152 catalog_manager.cc:5484] T d332a5e31e21465499833f960bf57e9b P 6117e90930294453ada6ea1fcca62663 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6117e90930294453ada6ea1fcca62663 (127.29.106.131). New cstate: current_term: 1 leader_uuid: "6117e90930294453ada6ea1fcca62663" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6117e90930294453ada6ea1fcca62663" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34645 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:11.332742 30122 tablet_server.cc:178] TabletServer@127.29.106.129:0 shutting down...
I20240502 07:58:11.338701 30122 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 07:58:11.339067 30122 tablet_replica.cc:330] T 08bdbe000f484a47a3cf825487fa498d P 3b7cbb0b145c46ad8ab5148fee0feed7: stopping tablet replica
I20240502 07:58:11.339409 30122 raft_consensus.cc:2229] T 08bdbe000f484a47a3cf825487fa498d P 3b7cbb0b145c46ad8ab5148fee0feed7 [term 1 LEADER]: Raft consensus shutting down.
I20240502 07:58:11.339587 30122 raft_consensus.cc:2258] T 08bdbe000f484a47a3cf825487fa498d P 3b7cbb0b145c46ad8ab5148fee0feed7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:11.341039 30122 tablet_replica.cc:330] T c05ae203ae1f463590579bfe11bf7932 P 3b7cbb0b145c46ad8ab5148fee0feed7: stopping tablet replica
I20240502 07:58:11.341156 30122 raft_consensus.cc:2229] T c05ae203ae1f463590579bfe11bf7932 P 3b7cbb0b145c46ad8ab5148fee0feed7 [term 1 LEADER]: Raft consensus shutting down.
I20240502 07:58:11.341266 30122 raft_consensus.cc:2258] T c05ae203ae1f463590579bfe11bf7932 P 3b7cbb0b145c46ad8ab5148fee0feed7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:11.357030 30122 tablet_server.cc:194] TabletServer@127.29.106.129:0 shutdown complete.
I20240502 07:58:11.362448 30122 tablet_server.cc:178] TabletServer@127.29.106.130:0 shutting down...
I20240502 07:58:11.367009 30122 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 07:58:11.367477 30122 tablet_replica.cc:330] T 7851822f3fbc449e9c463dadeaf3f92d P 67e75196581a47cfa5befc23e8bce1ee: stopping tablet replica
I20240502 07:58:11.367646 30122 raft_consensus.cc:2229] T 7851822f3fbc449e9c463dadeaf3f92d P 67e75196581a47cfa5befc23e8bce1ee [term 1 LEADER]: Raft consensus shutting down.
I20240502 07:58:11.367821 30122 raft_consensus.cc:2258] T 7851822f3fbc449e9c463dadeaf3f92d P 67e75196581a47cfa5befc23e8bce1ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:11.382066 30122 tablet_server.cc:194] TabletServer@127.29.106.130:0 shutdown complete.
I20240502 07:58:11.384728 30122 tablet_server.cc:178] TabletServer@127.29.106.131:0 shutting down...
I20240502 07:58:11.388706 30122 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 07:58:11.389119 30122 tablet_replica.cc:330] T d332a5e31e21465499833f960bf57e9b P 6117e90930294453ada6ea1fcca62663: stopping tablet replica
I20240502 07:58:11.389272 30122 raft_consensus.cc:2229] T d332a5e31e21465499833f960bf57e9b P 6117e90930294453ada6ea1fcca62663 [term 1 LEADER]: Raft consensus shutting down.
I20240502 07:58:11.389431 30122 raft_consensus.cc:2258] T d332a5e31e21465499833f960bf57e9b P 6117e90930294453ada6ea1fcca62663 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:11.403630 30122 tablet_server.cc:194] TabletServer@127.29.106.131:0 shutdown complete.
I20240502 07:58:11.406416 30122 master.cc:537] Master@127.29.106.190:41955 shutting down...
I20240502 07:58:11.411128 30122 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 07:58:11.411487 30122 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:11.411563 30122 tablet_replica.cc:330] T 00000000000000000000000000000000 P fe45358c9b3341f099e7141930e800f4: stopping tablet replica
I20240502 07:58:11.425814 30122 master.cc:552] Master@127.29.106.190:41955 shutdown complete.
I20240502 07:58:11.429399 30122 master.cc:537] Master@127.29.106.189:39967 shutting down...
I20240502 07:58:11.433812 30122 raft_consensus.cc:2229] T 00000000000000000000000000000000 P af6faa56b7bb4af081b158f39db0d9bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:11.434048 30122 raft_consensus.cc:2258] T 00000000000000000000000000000000 P af6faa56b7bb4af081b158f39db0d9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:11.434134 30122 tablet_replica.cc:330] T 00000000000000000000000000000000 P af6faa56b7bb4af081b158f39db0d9bd: stopping tablet replica
I20240502 07:58:11.447839 30122 master.cc:552] Master@127.29.106.189:39967 shutdown complete.
I20240502 07:58:11.451107 30122 master.cc:537] Master@127.29.106.188:35115 shutting down...
I20240502 07:58:11.455855 30122 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3da78873fde941b2bc323b892c7191e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:11.456089 30122 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3da78873fde941b2bc323b892c7191e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:11.456341 30122 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3da78873fde941b2bc323b892c7191e0: stopping tablet replica
I20240502 07:58:11.469528 30122 master.cc:552] Master@127.29.106.188:35115 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (2619 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20240502 07:58:11.478400 30122 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.106.190:39861
I20240502 07:58:11.478749 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:11.480866 30666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:11.481626 30669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:11.482291 30667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:11.482992 30122 server_base.cc:1034] running on GCE node
I20240502 07:58:11.483157 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:58:11.483196 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:58:11.483211 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636691483211 us; error 0 us; skew 500 ppm
I20240502 07:58:11.484227 30122 webserver.cc:460] Webserver started at http://127.29.106.190:35525/ using document root <none> and password file <none>
I20240502 07:58:11.484405 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:11.484457 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:11.484514 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:11.484881 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636688849691-30122-0/minicluster-data/master-0-root/instance:
uuid: "fe84c8508f9643309015ae12d6058dbb"
format_stamp: "Formatted at 2024-05-02 07:58:11 on dist-test-slave-m6l2"
I20240502 07:58:11.486411 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 07:58:11.487396 30674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:11.487821 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:11.487962 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636688849691-30122-0/minicluster-data/master-0-root
uuid: "fe84c8508f9643309015ae12d6058dbb"
format_stamp: "Formatted at 2024-05-02 07:58:11 on dist-test-slave-m6l2"
I20240502 07:58:11.488067 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636688849691-30122-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636688849691-30122-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636688849691-30122-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:11.501583 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:11.502036 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:11.506376 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.190:39861
I20240502 07:58:11.509678 30726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.190:39861 every 8 connection(s)
I20240502 07:58:11.510044 30727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:11.511757 30727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe84c8508f9643309015ae12d6058dbb: Bootstrap starting.
I20240502 07:58:11.512501 30727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe84c8508f9643309015ae12d6058dbb: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:11.513731 30727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe84c8508f9643309015ae12d6058dbb: No bootstrap required, opened a new log
I20240502 07:58:11.514137 30727 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe84c8508f9643309015ae12d6058dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe84c8508f9643309015ae12d6058dbb" member_type: VOTER }
I20240502 07:58:11.514241 30727 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe84c8508f9643309015ae12d6058dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:11.514263 30727 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe84c8508f9643309015ae12d6058dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe84c8508f9643309015ae12d6058dbb, State: Initialized, Role: FOLLOWER
I20240502 07:58:11.514376 30727 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe84c8508f9643309015ae12d6058dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe84c8508f9643309015ae12d6058dbb" member_type: VOTER }
I20240502 07:58:11.514474 30727 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe84c8508f9643309015ae12d6058dbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:58:11.514501 30727 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe84c8508f9643309015ae12d6058dbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:58:11.514531 30727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe84c8508f9643309015ae12d6058dbb [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:11.515218 30727 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe84c8508f9643309015ae12d6058dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe84c8508f9643309015ae12d6058dbb" member_type: VOTER }
I20240502 07:58:11.515328 30727 leader_election.cc:302] T 00000000000000000000000000000000 P fe84c8508f9643309015ae12d6058dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe84c8508f9643309015ae12d6058dbb; no voters:
I20240502 07:58:11.515504 30727 leader_election.cc:288] T 00000000000000000000000000000000 P fe84c8508f9643309015ae12d6058dbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:58:11.515758 30730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe84c8508f9643309015ae12d6058dbb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:11.515981 30727 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe84c8508f9643309015ae12d6058dbb [sys.catalog]: configured and running, proceeding with master startup.
I20240502 07:58:11.516059 30730 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe84c8508f9643309015ae12d6058dbb [term 1 LEADER]: Becoming Leader. State: Replica: fe84c8508f9643309015ae12d6058dbb, State: Running, Role: LEADER
I20240502 07:58:11.516242 30730 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe84c8508f9643309015ae12d6058dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe84c8508f9643309015ae12d6058dbb" member_type: VOTER }
I20240502 07:58:11.516844 30732 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe84c8508f9643309015ae12d6058dbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe84c8508f9643309015ae12d6058dbb. Latest consensus state: current_term: 1 leader_uuid: "fe84c8508f9643309015ae12d6058dbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe84c8508f9643309015ae12d6058dbb" member_type: VOTER } }
I20240502 07:58:11.516945 30732 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe84c8508f9643309015ae12d6058dbb [sys.catalog]: This master's current role is: LEADER
I20240502 07:58:11.517328 30739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 07:58:11.517599 30731 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe84c8508f9643309015ae12d6058dbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe84c8508f9643309015ae12d6058dbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe84c8508f9643309015ae12d6058dbb" member_type: VOTER } }
I20240502 07:58:11.517721 30731 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe84c8508f9643309015ae12d6058dbb [sys.catalog]: This master's current role is: LEADER
I20240502 07:58:11.518536 30739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 07:58:11.518791 30122 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 07:58:11.520898 30739 catalog_manager.cc:1347] Generated new cluster ID: 0fb232c9c1584167a6c8398b53eedc2c
I20240502 07:58:11.520972 30739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 07:58:11.533427 30739 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 07:58:11.534094 30739 catalog_manager.cc:1504] Loading token signing keys...
I20240502 07:58:11.543088 30739 catalog_manager.cc:5857] T 00000000000000000000000000000000 P fe84c8508f9643309015ae12d6058dbb: Generated new TSK 0
I20240502 07:58:11.543318 30739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 07:58:11.551728 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:11.554684 30748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:11.555429 30749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:11.556108 30751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:11.556672 30122 server_base.cc:1034] running on GCE node
I20240502 07:58:11.556912 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:58:11.556988 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:58:11.557024 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636691557022 us; error 0 us; skew 500 ppm
I20240502 07:58:11.558208 30122 webserver.cc:460] Webserver started at http://127.29.106.129:46225/ using document root <none> and password file <none>
I20240502 07:58:11.558398 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:11.558456 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:11.558625 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:11.559083 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636688849691-30122-0/minicluster-data/ts-0-root/instance:
uuid: "782a2eeace8d4dc68f55d03c1c745d3c"
format_stamp: "Formatted at 2024-05-02 07:58:11 on dist-test-slave-m6l2"
I20240502 07:58:11.560709 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 07:58:11.561728 30756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:11.562115 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:11.562197 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636688849691-30122-0/minicluster-data/ts-0-root
uuid: "782a2eeace8d4dc68f55d03c1c745d3c"
format_stamp: "Formatted at 2024-05-02 07:58:11 on dist-test-slave-m6l2"
I20240502 07:58:11.562287 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636688849691-30122-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636688849691-30122-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636688849691-30122-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:11.574311 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:11.574776 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:11.575085 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:11.575626 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:11.575680 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:11.575732 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:11.575753 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:11.582053 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.129:41727
I20240502 07:58:11.582707 30819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.129:41727 every 8 connection(s)
I20240502 07:58:11.583189 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:11.586649 30824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:11.587241 30825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:11.587852 30122 server_base.cc:1034] running on GCE node
W20240502 07:58:11.588388 30827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:11.588680 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:58:11.588738 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:58:11.588761 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636691588761 us; error 0 us; skew 500 ppm
I20240502 07:58:11.589990 30122 webserver.cc:460] Webserver started at http://127.29.106.130:43007/ using document root <none> and password file <none>
I20240502 07:58:11.590126 30820 heartbeater.cc:347] Connected to a master server at 127.29.106.190:39861
I20240502 07:58:11.590178 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:11.590231 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:11.590268 30820 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:11.590317 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:11.590569 30820 heartbeater.cc:511] Master 127.29.106.190:39861 requested a full tablet report, sending...
I20240502 07:58:11.590770 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636688849691-30122-0/minicluster-data/ts-1-root/instance:
uuid: "615b7a2a3e424b8ca8b1182212a0e945"
format_stamp: "Formatted at 2024-05-02 07:58:11 on dist-test-slave-m6l2"
I20240502 07:58:11.591410 30691 ts_manager.cc:194] Registered new tserver with Master: 782a2eeace8d4dc68f55d03c1c745d3c (127.29.106.129:41727)
I20240502 07:58:11.592494 30691 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42670
I20240502 07:58:11.592711 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240502 07:58:11.594090 30832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:11.594450 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:11.594529 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636688849691-30122-0/minicluster-data/ts-1-root
uuid: "615b7a2a3e424b8ca8b1182212a0e945"
format_stamp: "Formatted at 2024-05-02 07:58:11 on dist-test-slave-m6l2"
I20240502 07:58:11.594610 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636688849691-30122-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636688849691-30122-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636688849691-30122-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:11.604127 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:11.604576 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:11.604880 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:11.605370 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:11.605412 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:11.605463 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:11.605495 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:11.610062 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.130:34139
I20240502 07:58:11.610136 30895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.130:34139 every 8 connection(s)
I20240502 07:58:11.611083 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:11.614069 30900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:11.614320 30899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:11.614934 30902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:11.615751 30122 server_base.cc:1034] running on GCE node
I20240502 07:58:11.616114 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:58:11.616174 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:58:11.616226 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636691616225 us; error 0 us; skew 500 ppm
I20240502 07:58:11.616673 30896 heartbeater.cc:347] Connected to a master server at 127.29.106.190:39861
I20240502 07:58:11.616842 30896 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:11.617204 30896 heartbeater.cc:511] Master 127.29.106.190:39861 requested a full tablet report, sending...
I20240502 07:58:11.617519 30122 webserver.cc:460] Webserver started at http://127.29.106.131:32779/ using document root <none> and password file <none>
I20240502 07:58:11.617707 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:11.617769 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:11.617841 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:11.618043 30691 ts_manager.cc:194] Registered new tserver with Master: 615b7a2a3e424b8ca8b1182212a0e945 (127.29.106.130:34139)
I20240502 07:58:11.618384 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636688849691-30122-0/minicluster-data/ts-2-root/instance:
uuid: "c9ffe0c529954a0f88581546fc6573d3"
format_stamp: "Formatted at 2024-05-02 07:58:11 on dist-test-slave-m6l2"
I20240502 07:58:11.618677 30691 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42684
I20240502 07:58:11.620361 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:58:11.621596 30907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:11.621971 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:11.622056 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636688849691-30122-0/minicluster-data/ts-2-root
uuid: "c9ffe0c529954a0f88581546fc6573d3"
format_stamp: "Formatted at 2024-05-02 07:58:11 on dist-test-slave-m6l2"
I20240502 07:58:11.622149 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636688849691-30122-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636688849691-30122-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636688849691-30122-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:11.631430 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:11.632067 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:11.632563 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:11.633188 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:11.633255 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:11.633306 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:11.633343 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:11.639948 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.131:34467
I20240502 07:58:11.640043 30970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.131:34467 every 8 connection(s)
I20240502 07:58:11.645823 30971 heartbeater.cc:347] Connected to a master server at 127.29.106.190:39861
I20240502 07:58:11.646039 30971 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:11.646355 30971 heartbeater.cc:511] Master 127.29.106.190:39861 requested a full tablet report, sending...
I20240502 07:58:11.647130 30691 ts_manager.cc:194] Registered new tserver with Master: c9ffe0c529954a0f88581546fc6573d3 (127.29.106.131:34467)
I20240502 07:58:11.647480 30122 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006937183s
I20240502 07:58:11.647895 30691 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42686
I20240502 07:58:12.593773 30820 heartbeater.cc:503] Master 127.29.106.190:39861 was elected leader, sending a full tablet report...
I20240502 07:58:12.619961 30896 heartbeater.cc:503] Master 127.29.106.190:39861 was elected leader, sending a full tablet report...
I20240502 07:58:12.649225 30971 heartbeater.cc:503] Master 127.29.106.190:39861 was elected leader, sending a full tablet report...
I20240502 07:58:12.674563 30122 test_util.cc:268] Using random seed: -151234560
I20240502 07:58:12.684813 30691 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42702:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 07:58:12.685420 30691 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 07:58:12.700162 30779 tablet_service.cc:1467] Processing CreateTablet for tablet 98055797532e45b3a72d6ef4f278bd76 (DEFAULT_TABLE table=test-workload [id=8e41a0747d0044079c344f69a9a16573]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 07:58:12.700634 30777 tablet_service.cc:1467] Processing CreateTablet for tablet 229fec0abb91487582fad4da9f264ef5 (DEFAULT_TABLE table=test-workload [id=8e41a0747d0044079c344f69a9a16573]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 07:58:12.700699 30779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98055797532e45b3a72d6ef4f278bd76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:12.700973 30777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 229fec0abb91487582fad4da9f264ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:12.701268 30860 tablet_service.cc:1467] Processing CreateTablet for tablet e6509f5318594a9d8e8d8833202652e7 (DEFAULT_TABLE table=test-workload [id=8e41a0747d0044079c344f69a9a16573]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 07:58:12.701534 30860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6509f5318594a9d8e8d8833202652e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:12.701485 30781 tablet_service.cc:1467] Processing CreateTablet for tablet d5e959b5de9f4f289f92c31cbcb8234a (DEFAULT_TABLE table=test-workload [id=8e41a0747d0044079c344f69a9a16573]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 07:58:12.701809 30781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5e959b5de9f4f289f92c31cbcb8234a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:12.703943 30935 tablet_service.cc:1467] Processing CreateTablet for tablet e6509f5318594a9d8e8d8833202652e7 (DEFAULT_TABLE table=test-workload [id=8e41a0747d0044079c344f69a9a16573]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 07:58:12.703953 30859 tablet_service.cc:1467] Processing CreateTablet for tablet 321c5b918bc14901908bf94c28e0d4e3 (DEFAULT_TABLE table=test-workload [id=8e41a0747d0044079c344f69a9a16573]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 07:58:12.704232 30935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6509f5318594a9d8e8d8833202652e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:12.704165 30858 tablet_service.cc:1467] Processing CreateTablet for tablet 46c0c7f5bb734a19a515bf0869de62d6 (DEFAULT_TABLE table=test-workload [id=8e41a0747d0044079c344f69a9a16573]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 07:58:12.704368 30859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 321c5b918bc14901908bf94c28e0d4e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:12.704517 30783 tablet_service.cc:1467] Processing CreateTablet for tablet 321c5b918bc14901908bf94c28e0d4e3 (DEFAULT_TABLE table=test-workload [id=8e41a0747d0044079c344f69a9a16573]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 07:58:12.704648 30855 tablet_service.cc:1467] Processing CreateTablet for tablet d5e959b5de9f4f289f92c31cbcb8234a (DEFAULT_TABLE table=test-workload [id=8e41a0747d0044079c344f69a9a16573]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 07:58:12.704752 30783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 321c5b918bc14901908bf94c28e0d4e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:12.704874 30855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5e959b5de9f4f289f92c31cbcb8234a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:12.707000 30933 tablet_service.cc:1467] Processing CreateTablet for tablet 46c0c7f5bb734a19a515bf0869de62d6 (DEFAULT_TABLE table=test-workload [id=8e41a0747d0044079c344f69a9a16573]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 07:58:12.707170 30784 tablet_service.cc:1467] Processing CreateTablet for tablet e6509f5318594a9d8e8d8833202652e7 (DEFAULT_TABLE table=test-workload [id=8e41a0747d0044079c344f69a9a16573]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 07:58:12.707257 30933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46c0c7f5bb734a19a515bf0869de62d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:12.707423 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6509f5318594a9d8e8d8833202652e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:12.707494 30857 tablet_service.cc:1467] Processing CreateTablet for tablet 3175330f9b374e559549707d1107306b (DEFAULT_TABLE table=test-workload [id=8e41a0747d0044079c344f69a9a16573]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 07:58:12.707722 30857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3175330f9b374e559549707d1107306b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:12.708693 30782 tablet_service.cc:1467] Processing CreateTablet for tablet 46c0c7f5bb734a19a515bf0869de62d6 (DEFAULT_TABLE table=test-workload [id=8e41a0747d0044079c344f69a9a16573]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 07:58:12.708981 30782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46c0c7f5bb734a19a515bf0869de62d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:12.709787 30931 tablet_service.cc:1467] Processing CreateTablet for tablet 3175330f9b374e559549707d1107306b (DEFAULT_TABLE table=test-workload [id=8e41a0747d0044079c344f69a9a16573]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 07:58:12.710045 30931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3175330f9b374e559549707d1107306b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:12.710208 30856 tablet_service.cc:1467] Processing CreateTablet for tablet 98055797532e45b3a72d6ef4f278bd76 (DEFAULT_TABLE table=test-workload [id=8e41a0747d0044079c344f69a9a16573]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 07:58:12.710476 30856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98055797532e45b3a72d6ef4f278bd76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:12.712347 30928 tablet_service.cc:1467] Processing CreateTablet for tablet 229fec0abb91487582fad4da9f264ef5 (DEFAULT_TABLE table=test-workload [id=8e41a0747d0044079c344f69a9a16573]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 07:58:12.712528 30854 tablet_service.cc:1467] Processing CreateTablet for tablet d37ebaee499a4240a6ecfb02aab87b9a (DEFAULT_TABLE table=test-workload [id=8e41a0747d0044079c344f69a9a16573]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 07:58:12.712517 30929 tablet_service.cc:1467] Processing CreateTablet for tablet d37ebaee499a4240a6ecfb02aab87b9a (DEFAULT_TABLE table=test-workload [id=8e41a0747d0044079c344f69a9a16573]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 07:58:12.712762 30928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 229fec0abb91487582fad4da9f264ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:12.712726 30853 tablet_service.cc:1467] Processing CreateTablet for tablet 229fec0abb91487582fad4da9f264ef5 (DEFAULT_TABLE table=test-workload [id=8e41a0747d0044079c344f69a9a16573]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 07:58:12.712834 30854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37ebaee499a4240a6ecfb02aab87b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:12.712972 30853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 229fec0abb91487582fad4da9f264ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:12.713001 30929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37ebaee499a4240a6ecfb02aab87b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:12.700227 30778 tablet_service.cc:1467] Processing CreateTablet for tablet d37ebaee499a4240a6ecfb02aab87b9a (DEFAULT_TABLE table=test-workload [id=8e41a0747d0044079c344f69a9a16573]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 07:58:12.713577 30778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37ebaee499a4240a6ecfb02aab87b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:12.715581 30934 tablet_service.cc:1467] Processing CreateTablet for tablet 321c5b918bc14901908bf94c28e0d4e3 (DEFAULT_TABLE table=test-workload [id=8e41a0747d0044079c344f69a9a16573]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 07:58:12.715895 30934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 321c5b918bc14901908bf94c28e0d4e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:12.716146 30992 tablet_bootstrap.cc:492] T e6509f5318594a9d8e8d8833202652e7 P 615b7a2a3e424b8ca8b1182212a0e945: Bootstrap starting.
I20240502 07:58:12.717525 30992 tablet_bootstrap.cc:654] T e6509f5318594a9d8e8d8833202652e7 P 615b7a2a3e424b8ca8b1182212a0e945: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:12.718225 30993 tablet_bootstrap.cc:492] T e6509f5318594a9d8e8d8833202652e7 P c9ffe0c529954a0f88581546fc6573d3: Bootstrap starting.
I20240502 07:58:12.718627 30932 tablet_service.cc:1467] Processing CreateTablet for tablet d5e959b5de9f4f289f92c31cbcb8234a (DEFAULT_TABLE table=test-workload [id=8e41a0747d0044079c344f69a9a16573]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 07:58:12.718943 30932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5e959b5de9f4f289f92c31cbcb8234a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:12.718999 30930 tablet_service.cc:1467] Processing CreateTablet for tablet 98055797532e45b3a72d6ef4f278bd76 (DEFAULT_TABLE table=test-workload [id=8e41a0747d0044079c344f69a9a16573]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 07:58:12.719249 30930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98055797532e45b3a72d6ef4f278bd76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:12.719247 30993 tablet_bootstrap.cc:654] T e6509f5318594a9d8e8d8833202652e7 P c9ffe0c529954a0f88581546fc6573d3: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:12.701265 30780 tablet_service.cc:1467] Processing CreateTablet for tablet 3175330f9b374e559549707d1107306b (DEFAULT_TABLE table=test-workload [id=8e41a0747d0044079c344f69a9a16573]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 07:58:12.719921 30780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3175330f9b374e559549707d1107306b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:12.720723 30993 tablet_bootstrap.cc:492] T e6509f5318594a9d8e8d8833202652e7 P c9ffe0c529954a0f88581546fc6573d3: No bootstrap required, opened a new log
I20240502 07:58:12.720813 30993 ts_tablet_manager.cc:1367] T e6509f5318594a9d8e8d8833202652e7 P c9ffe0c529954a0f88581546fc6573d3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:12.721280 30993 raft_consensus.cc:348] T e6509f5318594a9d8e8d8833202652e7 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.721382 30993 raft_consensus.cc:374] T e6509f5318594a9d8e8d8833202652e7 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:12.721417 30993 raft_consensus.cc:729] T e6509f5318594a9d8e8d8833202652e7 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9ffe0c529954a0f88581546fc6573d3, State: Initialized, Role: FOLLOWER
I20240502 07:58:12.721446 30858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46c0c7f5bb734a19a515bf0869de62d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:12.721558 30993 consensus_queue.cc:260] T e6509f5318594a9d8e8d8833202652e7 P c9ffe0c529954a0f88581546fc6573d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.721721 30992 tablet_bootstrap.cc:492] T e6509f5318594a9d8e8d8833202652e7 P 615b7a2a3e424b8ca8b1182212a0e945: No bootstrap required, opened a new log
I20240502 07:58:12.721812 30992 ts_tablet_manager.cc:1367] T e6509f5318594a9d8e8d8833202652e7 P 615b7a2a3e424b8ca8b1182212a0e945: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 07:58:12.721913 30993 ts_tablet_manager.cc:1398] T e6509f5318594a9d8e8d8833202652e7 P c9ffe0c529954a0f88581546fc6573d3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:12.722159 30993 tablet_bootstrap.cc:492] T 46c0c7f5bb734a19a515bf0869de62d6 P c9ffe0c529954a0f88581546fc6573d3: Bootstrap starting.
I20240502 07:58:12.722375 30992 raft_consensus.cc:348] T e6509f5318594a9d8e8d8833202652e7 P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.722488 30992 raft_consensus.cc:374] T e6509f5318594a9d8e8d8833202652e7 P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:12.722537 30992 raft_consensus.cc:729] T e6509f5318594a9d8e8d8833202652e7 P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615b7a2a3e424b8ca8b1182212a0e945, State: Initialized, Role: FOLLOWER
I20240502 07:58:12.722721 30992 consensus_queue.cc:260] T e6509f5318594a9d8e8d8833202652e7 P 615b7a2a3e424b8ca8b1182212a0e945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.723151 30993 tablet_bootstrap.cc:654] T 46c0c7f5bb734a19a515bf0869de62d6 P c9ffe0c529954a0f88581546fc6573d3: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:12.723191 30992 ts_tablet_manager.cc:1398] T e6509f5318594a9d8e8d8833202652e7 P 615b7a2a3e424b8ca8b1182212a0e945: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:58:12.724766 30993 tablet_bootstrap.cc:492] T 46c0c7f5bb734a19a515bf0869de62d6 P c9ffe0c529954a0f88581546fc6573d3: No bootstrap required, opened a new log
I20240502 07:58:12.724869 30993 ts_tablet_manager.cc:1367] T 46c0c7f5bb734a19a515bf0869de62d6 P c9ffe0c529954a0f88581546fc6573d3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:12.725142 30991 tablet_bootstrap.cc:492] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c: Bootstrap starting.
I20240502 07:58:12.725344 30993 raft_consensus.cc:348] T 46c0c7f5bb734a19a515bf0869de62d6 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.725484 30993 raft_consensus.cc:374] T 46c0c7f5bb734a19a515bf0869de62d6 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:12.725533 30993 raft_consensus.cc:729] T 46c0c7f5bb734a19a515bf0869de62d6 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9ffe0c529954a0f88581546fc6573d3, State: Initialized, Role: FOLLOWER
I20240502 07:58:12.725680 30993 consensus_queue.cc:260] T 46c0c7f5bb734a19a515bf0869de62d6 P c9ffe0c529954a0f88581546fc6573d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.725952 30993 ts_tablet_manager.cc:1398] T 46c0c7f5bb734a19a515bf0869de62d6 P c9ffe0c529954a0f88581546fc6573d3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:12.726187 30993 tablet_bootstrap.cc:492] T 3175330f9b374e559549707d1107306b P c9ffe0c529954a0f88581546fc6573d3: Bootstrap starting.
I20240502 07:58:12.726291 30991 tablet_bootstrap.cc:654] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:12.725962 30992 tablet_bootstrap.cc:492] T 321c5b918bc14901908bf94c28e0d4e3 P 615b7a2a3e424b8ca8b1182212a0e945: Bootstrap starting.
I20240502 07:58:12.727241 30993 tablet_bootstrap.cc:654] T 3175330f9b374e559549707d1107306b P c9ffe0c529954a0f88581546fc6573d3: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:12.727609 30991 tablet_bootstrap.cc:492] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c: No bootstrap required, opened a new log
I20240502 07:58:12.727691 30991 ts_tablet_manager.cc:1367] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:12.727834 30992 tablet_bootstrap.cc:654] T 321c5b918bc14901908bf94c28e0d4e3 P 615b7a2a3e424b8ca8b1182212a0e945: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:12.728220 30991 raft_consensus.cc:348] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.728384 30991 raft_consensus.cc:374] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:12.728422 30991 raft_consensus.cc:729] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 782a2eeace8d4dc68f55d03c1c745d3c, State: Initialized, Role: FOLLOWER
I20240502 07:58:12.728461 30993 tablet_bootstrap.cc:492] T 3175330f9b374e559549707d1107306b P c9ffe0c529954a0f88581546fc6573d3: No bootstrap required, opened a new log
I20240502 07:58:12.728552 30993 ts_tablet_manager.cc:1367] T 3175330f9b374e559549707d1107306b P c9ffe0c529954a0f88581546fc6573d3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:12.728567 30991 consensus_queue.cc:260] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.728938 30991 ts_tablet_manager.cc:1398] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:58:12.729002 30993 raft_consensus.cc:348] T 3175330f9b374e559549707d1107306b P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.729130 30993 raft_consensus.cc:374] T 3175330f9b374e559549707d1107306b P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:12.729182 30993 raft_consensus.cc:729] T 3175330f9b374e559549707d1107306b P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9ffe0c529954a0f88581546fc6573d3, State: Initialized, Role: FOLLOWER
I20240502 07:58:12.729333 30993 consensus_queue.cc:260] T 3175330f9b374e559549707d1107306b P c9ffe0c529954a0f88581546fc6573d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.729631 30993 ts_tablet_manager.cc:1398] T 3175330f9b374e559549707d1107306b P c9ffe0c529954a0f88581546fc6573d3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:12.729135 30991 tablet_bootstrap.cc:492] T d5e959b5de9f4f289f92c31cbcb8234a P 782a2eeace8d4dc68f55d03c1c745d3c: Bootstrap starting.
I20240502 07:58:12.729821 30993 tablet_bootstrap.cc:492] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3: Bootstrap starting.
I20240502 07:58:12.730762 30991 tablet_bootstrap.cc:654] T d5e959b5de9f4f289f92c31cbcb8234a P 782a2eeace8d4dc68f55d03c1c745d3c: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:12.730840 30993 tablet_bootstrap.cc:654] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:12.731976 30993 tablet_bootstrap.cc:492] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3: No bootstrap required, opened a new log
I20240502 07:58:12.732065 30993 ts_tablet_manager.cc:1367] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:12.732575 30992 tablet_bootstrap.cc:492] T 321c5b918bc14901908bf94c28e0d4e3 P 615b7a2a3e424b8ca8b1182212a0e945: No bootstrap required, opened a new log
I20240502 07:58:12.732575 30993 raft_consensus.cc:348] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.732702 30992 ts_tablet_manager.cc:1367] T 321c5b918bc14901908bf94c28e0d4e3 P 615b7a2a3e424b8ca8b1182212a0e945: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 07:58:12.732720 30993 raft_consensus.cc:374] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:12.732751 30993 raft_consensus.cc:729] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9ffe0c529954a0f88581546fc6573d3, State: Initialized, Role: FOLLOWER
I20240502 07:58:12.732870 30993 consensus_queue.cc:260] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.733166 30993 ts_tablet_manager.cc:1398] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:12.733353 30993 tablet_bootstrap.cc:492] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3: Bootstrap starting.
I20240502 07:58:12.733939 30992 raft_consensus.cc:348] T 321c5b918bc14901908bf94c28e0d4e3 P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.734054 30992 raft_consensus.cc:374] T 321c5b918bc14901908bf94c28e0d4e3 P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:12.734360 30992 raft_consensus.cc:729] T 321c5b918bc14901908bf94c28e0d4e3 P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615b7a2a3e424b8ca8b1182212a0e945, State: Initialized, Role: FOLLOWER
I20240502 07:58:12.734321 30993 tablet_bootstrap.cc:654] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:12.734505 30992 consensus_queue.cc:260] T 321c5b918bc14901908bf94c28e0d4e3 P 615b7a2a3e424b8ca8b1182212a0e945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.734889 30992 ts_tablet_manager.cc:1398] T 321c5b918bc14901908bf94c28e0d4e3 P 615b7a2a3e424b8ca8b1182212a0e945: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:12.735070 30992 tablet_bootstrap.cc:492] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945: Bootstrap starting.
I20240502 07:58:12.735610 30991 tablet_bootstrap.cc:492] T d5e959b5de9f4f289f92c31cbcb8234a P 782a2eeace8d4dc68f55d03c1c745d3c: No bootstrap required, opened a new log
I20240502 07:58:12.735706 30991 ts_tablet_manager.cc:1367] T d5e959b5de9f4f289f92c31cbcb8234a P 782a2eeace8d4dc68f55d03c1c745d3c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 07:58:12.736011 30992 tablet_bootstrap.cc:654] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:12.736157 30991 raft_consensus.cc:348] T d5e959b5de9f4f289f92c31cbcb8234a P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.736516 30991 raft_consensus.cc:374] T d5e959b5de9f4f289f92c31cbcb8234a P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:12.736585 30991 raft_consensus.cc:729] T d5e959b5de9f4f289f92c31cbcb8234a P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 782a2eeace8d4dc68f55d03c1c745d3c, State: Initialized, Role: FOLLOWER
I20240502 07:58:12.736707 30991 consensus_queue.cc:260] T d5e959b5de9f4f289f92c31cbcb8234a P 782a2eeace8d4dc68f55d03c1c745d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.737046 30991 ts_tablet_manager.cc:1398] T d5e959b5de9f4f289f92c31cbcb8234a P 782a2eeace8d4dc68f55d03c1c745d3c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:12.737269 30991 tablet_bootstrap.cc:492] T 321c5b918bc14901908bf94c28e0d4e3 P 782a2eeace8d4dc68f55d03c1c745d3c: Bootstrap starting.
I20240502 07:58:12.738157 30991 tablet_bootstrap.cc:654] T 321c5b918bc14901908bf94c28e0d4e3 P 782a2eeace8d4dc68f55d03c1c745d3c: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:12.739543 30993 tablet_bootstrap.cc:492] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3: No bootstrap required, opened a new log
I20240502 07:58:12.739639 30993 ts_tablet_manager.cc:1367] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 07:58:12.740092 30993 raft_consensus.cc:348] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.740211 30993 raft_consensus.cc:374] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:12.740245 30993 raft_consensus.cc:729] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9ffe0c529954a0f88581546fc6573d3, State: Initialized, Role: FOLLOWER
I20240502 07:58:12.739645 30991 tablet_bootstrap.cc:492] T 321c5b918bc14901908bf94c28e0d4e3 P 782a2eeace8d4dc68f55d03c1c745d3c: No bootstrap required, opened a new log
I20240502 07:58:12.740474 30991 ts_tablet_manager.cc:1367] T 321c5b918bc14901908bf94c28e0d4e3 P 782a2eeace8d4dc68f55d03c1c745d3c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:12.740455 30993 consensus_queue.cc:260] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.740691 30993 ts_tablet_manager.cc:1398] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:12.740869 30993 tablet_bootstrap.cc:492] T d37ebaee499a4240a6ecfb02aab87b9a P c9ffe0c529954a0f88581546fc6573d3: Bootstrap starting.
I20240502 07:58:12.740947 30991 raft_consensus.cc:348] T 321c5b918bc14901908bf94c28e0d4e3 P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.741117 30991 raft_consensus.cc:374] T 321c5b918bc14901908bf94c28e0d4e3 P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:12.741153 30991 raft_consensus.cc:729] T 321c5b918bc14901908bf94c28e0d4e3 P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 782a2eeace8d4dc68f55d03c1c745d3c, State: Initialized, Role: FOLLOWER
I20240502 07:58:12.741297 30991 consensus_queue.cc:260] T 321c5b918bc14901908bf94c28e0d4e3 P 782a2eeace8d4dc68f55d03c1c745d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.741575 30991 ts_tablet_manager.cc:1398] T 321c5b918bc14901908bf94c28e0d4e3 P 782a2eeace8d4dc68f55d03c1c745d3c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:12.741763 30991 tablet_bootstrap.cc:492] T e6509f5318594a9d8e8d8833202652e7 P 782a2eeace8d4dc68f55d03c1c745d3c: Bootstrap starting.
I20240502 07:58:12.741845 30993 tablet_bootstrap.cc:654] T d37ebaee499a4240a6ecfb02aab87b9a P c9ffe0c529954a0f88581546fc6573d3: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:12.742642 30996 raft_consensus.cc:482] T 3175330f9b374e559549707d1107306b P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:12.742736 30996 raft_consensus.cc:504] T 3175330f9b374e559549707d1107306b P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.742854 30991 tablet_bootstrap.cc:654] T e6509f5318594a9d8e8d8833202652e7 P 782a2eeace8d4dc68f55d03c1c745d3c: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:12.743256 30996 leader_election.cc:288] T 3175330f9b374e559549707d1107306b P c9ffe0c529954a0f88581546fc6573d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 782a2eeace8d4dc68f55d03c1c745d3c (127.29.106.129:41727), 615b7a2a3e424b8ca8b1182212a0e945 (127.29.106.130:34139)
I20240502 07:58:12.744472 30993 tablet_bootstrap.cc:492] T d37ebaee499a4240a6ecfb02aab87b9a P c9ffe0c529954a0f88581546fc6573d3: No bootstrap required, opened a new log
I20240502 07:58:12.744578 30993 ts_tablet_manager.cc:1367] T d37ebaee499a4240a6ecfb02aab87b9a P c9ffe0c529954a0f88581546fc6573d3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:58:12.744738 30992 tablet_bootstrap.cc:492] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945: No bootstrap required, opened a new log
I20240502 07:58:12.744853 30992 ts_tablet_manager.cc:1367] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 07:58:12.745052 30993 raft_consensus.cc:348] T d37ebaee499a4240a6ecfb02aab87b9a P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.745162 30993 raft_consensus.cc:374] T d37ebaee499a4240a6ecfb02aab87b9a P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:12.745200 30993 raft_consensus.cc:729] T d37ebaee499a4240a6ecfb02aab87b9a P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9ffe0c529954a0f88581546fc6573d3, State: Initialized, Role: FOLLOWER
I20240502 07:58:12.745360 30993 consensus_queue.cc:260] T d37ebaee499a4240a6ecfb02aab87b9a P c9ffe0c529954a0f88581546fc6573d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.745513 30992 raft_consensus.cc:348] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.745612 30992 raft_consensus.cc:374] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:12.745659 30992 raft_consensus.cc:729] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615b7a2a3e424b8ca8b1182212a0e945, State: Initialized, Role: FOLLOWER
I20240502 07:58:12.745755 30992 consensus_queue.cc:260] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.746109 30992 ts_tablet_manager.cc:1398] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:12.746301 30992 tablet_bootstrap.cc:492] T 3175330f9b374e559549707d1107306b P 615b7a2a3e424b8ca8b1182212a0e945: Bootstrap starting.
I20240502 07:58:12.747326 30991 tablet_bootstrap.cc:492] T e6509f5318594a9d8e8d8833202652e7 P 782a2eeace8d4dc68f55d03c1c745d3c: No bootstrap required, opened a new log
I20240502 07:58:12.747417 30992 tablet_bootstrap.cc:654] T 3175330f9b374e559549707d1107306b P 615b7a2a3e424b8ca8b1182212a0e945: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:12.747447 30991 ts_tablet_manager.cc:1367] T e6509f5318594a9d8e8d8833202652e7 P 782a2eeace8d4dc68f55d03c1c745d3c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 07:58:12.748024 30991 raft_consensus.cc:348] T e6509f5318594a9d8e8d8833202652e7 P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.748160 30991 raft_consensus.cc:374] T e6509f5318594a9d8e8d8833202652e7 P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:12.748196 30991 raft_consensus.cc:729] T e6509f5318594a9d8e8d8833202652e7 P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 782a2eeace8d4dc68f55d03c1c745d3c, State: Initialized, Role: FOLLOWER
I20240502 07:58:12.748256 30996 raft_consensus.cc:482] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:12.748368 30991 consensus_queue.cc:260] T e6509f5318594a9d8e8d8833202652e7 P 782a2eeace8d4dc68f55d03c1c745d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.748391 30996 raft_consensus.cc:504] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.748752 30991 ts_tablet_manager.cc:1398] T e6509f5318594a9d8e8d8833202652e7 P 782a2eeace8d4dc68f55d03c1c745d3c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:58:12.748804 30996 leader_election.cc:288] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 782a2eeace8d4dc68f55d03c1c745d3c (127.29.106.129:41727), 615b7a2a3e424b8ca8b1182212a0e945 (127.29.106.130:34139)
I20240502 07:58:12.748967 30991 tablet_bootstrap.cc:492] T 46c0c7f5bb734a19a515bf0869de62d6 P 782a2eeace8d4dc68f55d03c1c745d3c: Bootstrap starting.
I20240502 07:58:12.749301 30993 ts_tablet_manager.cc:1398] T d37ebaee499a4240a6ecfb02aab87b9a P c9ffe0c529954a0f88581546fc6573d3: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 07:58:12.749492 30993 tablet_bootstrap.cc:492] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3: Bootstrap starting.
I20240502 07:58:12.750283 30991 tablet_bootstrap.cc:654] T 46c0c7f5bb734a19a515bf0869de62d6 P 782a2eeace8d4dc68f55d03c1c745d3c: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:12.750537 30993 tablet_bootstrap.cc:654] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:12.751570 30993 tablet_bootstrap.cc:492] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3: No bootstrap required, opened a new log
I20240502 07:58:12.751668 30993 ts_tablet_manager.cc:1367] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:12.752123 30993 raft_consensus.cc:348] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.752233 30993 raft_consensus.cc:374] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:12.752296 30993 raft_consensus.cc:729] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9ffe0c529954a0f88581546fc6573d3, State: Initialized, Role: FOLLOWER
I20240502 07:58:12.752440 30993 consensus_queue.cc:260] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.752696 30993 ts_tablet_manager.cc:1398] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:12.752858 30993 tablet_bootstrap.cc:492] T d5e959b5de9f4f289f92c31cbcb8234a P c9ffe0c529954a0f88581546fc6573d3: Bootstrap starting.
I20240502 07:58:12.753791 30993 tablet_bootstrap.cc:654] T d5e959b5de9f4f289f92c31cbcb8234a P c9ffe0c529954a0f88581546fc6573d3: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:12.751570 30991 tablet_bootstrap.cc:492] T 46c0c7f5bb734a19a515bf0869de62d6 P 782a2eeace8d4dc68f55d03c1c745d3c: No bootstrap required, opened a new log
I20240502 07:58:12.755084 30991 ts_tablet_manager.cc:1367] T 46c0c7f5bb734a19a515bf0869de62d6 P 782a2eeace8d4dc68f55d03c1c745d3c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 07:58:12.755501 30991 raft_consensus.cc:348] T 46c0c7f5bb734a19a515bf0869de62d6 P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.755618 30991 raft_consensus.cc:374] T 46c0c7f5bb734a19a515bf0869de62d6 P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:12.755652 30991 raft_consensus.cc:729] T 46c0c7f5bb734a19a515bf0869de62d6 P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 782a2eeace8d4dc68f55d03c1c745d3c, State: Initialized, Role: FOLLOWER
I20240502 07:58:12.755753 30794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3175330f9b374e559549707d1107306b" candidate_uuid: "c9ffe0c529954a0f88581546fc6573d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" is_pre_election: true
I20240502 07:58:12.756327 30793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "229fec0abb91487582fad4da9f264ef5" candidate_uuid: "c9ffe0c529954a0f88581546fc6573d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" is_pre_election: true
I20240502 07:58:12.756454 30793 raft_consensus.cc:2454] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9ffe0c529954a0f88581546fc6573d3 in term 0.
W20240502 07:58:12.756506 30911 leader_election.cc:341] T 3175330f9b374e559549707d1107306b P c9ffe0c529954a0f88581546fc6573d3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 782a2eeace8d4dc68f55d03c1c745d3c (127.29.106.129:41727): Illegal state: must be running to vote when last-logged opid is not known
I20240502 07:58:12.756784 30992 tablet_bootstrap.cc:492] T 3175330f9b374e559549707d1107306b P 615b7a2a3e424b8ca8b1182212a0e945: No bootstrap required, opened a new log
I20240502 07:58:12.756875 30992 ts_tablet_manager.cc:1367] T 3175330f9b374e559549707d1107306b P 615b7a2a3e424b8ca8b1182212a0e945: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20240502 07:58:12.756522 31002 raft_consensus.cc:482] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:12.756974 31002 raft_consensus.cc:504] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.755769 30991 consensus_queue.cc:260] T 46c0c7f5bb734a19a515bf0869de62d6 P 782a2eeace8d4dc68f55d03c1c745d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.757278 30992 raft_consensus.cc:348] T 3175330f9b374e559549707d1107306b P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.757544 30992 raft_consensus.cc:374] T 3175330f9b374e559549707d1107306b P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:12.757596 30992 raft_consensus.cc:729] T 3175330f9b374e559549707d1107306b P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615b7a2a3e424b8ca8b1182212a0e945, State: Initialized, Role: FOLLOWER
I20240502 07:58:12.757757 30992 consensus_queue.cc:260] T 3175330f9b374e559549707d1107306b P 615b7a2a3e424b8ca8b1182212a0e945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.758024 30991 ts_tablet_manager.cc:1398] T 46c0c7f5bb734a19a515bf0869de62d6 P 782a2eeace8d4dc68f55d03c1c745d3c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 07:58:12.759243 30991 tablet_bootstrap.cc:492] T 98055797532e45b3a72d6ef4f278bd76 P 782a2eeace8d4dc68f55d03c1c745d3c: Bootstrap starting.
I20240502 07:58:12.759557 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3175330f9b374e559549707d1107306b" candidate_uuid: "c9ffe0c529954a0f88581546fc6573d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615b7a2a3e424b8ca8b1182212a0e945" is_pre_election: true
I20240502 07:58:12.759621 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "229fec0abb91487582fad4da9f264ef5" candidate_uuid: "c9ffe0c529954a0f88581546fc6573d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615b7a2a3e424b8ca8b1182212a0e945" is_pre_election: true
I20240502 07:58:12.759706 30870 raft_consensus.cc:2454] T 3175330f9b374e559549707d1107306b P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9ffe0c529954a0f88581546fc6573d3 in term 0.
I20240502 07:58:12.760033 30909 leader_election.cc:302] T 3175330f9b374e559549707d1107306b P c9ffe0c529954a0f88581546fc6573d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 615b7a2a3e424b8ca8b1182212a0e945, c9ffe0c529954a0f88581546fc6573d3; no voters: 782a2eeace8d4dc68f55d03c1c745d3c
I20240502 07:58:12.760437 30996 raft_consensus.cc:2781] T 3175330f9b374e559549707d1107306b P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:12.760507 30996 raft_consensus.cc:482] T 3175330f9b374e559549707d1107306b P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:12.760558 30996 raft_consensus.cc:3037] T 3175330f9b374e559549707d1107306b P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:12.760903 30991 tablet_bootstrap.cc:654] T 98055797532e45b3a72d6ef4f278bd76 P 782a2eeace8d4dc68f55d03c1c745d3c: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:12.758600 30992 ts_tablet_manager.cc:1398] T 3175330f9b374e559549707d1107306b P 615b7a2a3e424b8ca8b1182212a0e945: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:12.758901 30911 leader_election.cc:302] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 782a2eeace8d4dc68f55d03c1c745d3c, c9ffe0c529954a0f88581546fc6573d3; no voters:
I20240502 07:58:12.757488 31002 leader_election.cc:288] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 615b7a2a3e424b8ca8b1182212a0e945 (127.29.106.130:34139), c9ffe0c529954a0f88581546fc6573d3 (127.29.106.131:34467)
I20240502 07:58:12.762218 30993 tablet_bootstrap.cc:492] T d5e959b5de9f4f289f92c31cbcb8234a P c9ffe0c529954a0f88581546fc6573d3: No bootstrap required, opened a new log
W20240502 07:58:12.762235 30909 leader_election.cc:341] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 615b7a2a3e424b8ca8b1182212a0e945 (127.29.106.130:34139): Illegal state: must be running to vote when last-logged opid is not known
I20240502 07:58:12.762336 30993 ts_tablet_manager.cc:1367] T d5e959b5de9f4f289f92c31cbcb8234a P c9ffe0c529954a0f88581546fc6573d3: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 07:58:12.762511 31020 raft_consensus.cc:2781] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:12.762553 31020 raft_consensus.cc:482] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:12.762595 31020 raft_consensus.cc:3037] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:12.762991 30992 tablet_bootstrap.cc:492] T d37ebaee499a4240a6ecfb02aab87b9a P 615b7a2a3e424b8ca8b1182212a0e945: Bootstrap starting.
I20240502 07:58:12.763002 30993 raft_consensus.cc:348] T d5e959b5de9f4f289f92c31cbcb8234a P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.763176 30993 raft_consensus.cc:374] T d5e959b5de9f4f289f92c31cbcb8234a P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:12.763211 30993 raft_consensus.cc:729] T d5e959b5de9f4f289f92c31cbcb8234a P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9ffe0c529954a0f88581546fc6573d3, State: Initialized, Role: FOLLOWER
I20240502 07:58:12.763327 30993 consensus_queue.cc:260] T d5e959b5de9f4f289f92c31cbcb8234a P c9ffe0c529954a0f88581546fc6573d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.763532 30991 tablet_bootstrap.cc:492] T 98055797532e45b3a72d6ef4f278bd76 P 782a2eeace8d4dc68f55d03c1c745d3c: No bootstrap required, opened a new log
I20240502 07:58:12.763639 30991 ts_tablet_manager.cc:1367] T 98055797532e45b3a72d6ef4f278bd76 P 782a2eeace8d4dc68f55d03c1c745d3c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 07:58:12.763737 30993 ts_tablet_manager.cc:1398] T d5e959b5de9f4f289f92c31cbcb8234a P c9ffe0c529954a0f88581546fc6573d3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:12.764102 30991 raft_consensus.cc:348] T 98055797532e45b3a72d6ef4f278bd76 P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.764200 30992 tablet_bootstrap.cc:654] T d37ebaee499a4240a6ecfb02aab87b9a P 615b7a2a3e424b8ca8b1182212a0e945: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:12.764235 30991 raft_consensus.cc:374] T 98055797532e45b3a72d6ef4f278bd76 P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:12.764343 30991 raft_consensus.cc:729] T 98055797532e45b3a72d6ef4f278bd76 P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 782a2eeace8d4dc68f55d03c1c745d3c, State: Initialized, Role: FOLLOWER
I20240502 07:58:12.764487 30991 consensus_queue.cc:260] T 98055797532e45b3a72d6ef4f278bd76 P 782a2eeace8d4dc68f55d03c1c745d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.764918 30991 ts_tablet_manager.cc:1398] T 98055797532e45b3a72d6ef4f278bd76 P 782a2eeace8d4dc68f55d03c1c745d3c: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 07:58:12.765122 30991 tablet_bootstrap.cc:492] T d37ebaee499a4240a6ecfb02aab87b9a P 782a2eeace8d4dc68f55d03c1c745d3c: Bootstrap starting.
I20240502 07:58:12.767302 30991 tablet_bootstrap.cc:654] T d37ebaee499a4240a6ecfb02aab87b9a P 782a2eeace8d4dc68f55d03c1c745d3c: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:12.767767 31022 raft_consensus.cc:482] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:12.767880 31022 raft_consensus.cc:504] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.768379 31022 leader_election.cc:288] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 782a2eeace8d4dc68f55d03c1c745d3c (127.29.106.129:41727), 615b7a2a3e424b8ca8b1182212a0e945 (127.29.106.130:34139)
I20240502 07:58:12.768693 30991 tablet_bootstrap.cc:492] T d37ebaee499a4240a6ecfb02aab87b9a P 782a2eeace8d4dc68f55d03c1c745d3c: No bootstrap required, opened a new log
I20240502 07:58:12.768783 30991 ts_tablet_manager.cc:1367] T d37ebaee499a4240a6ecfb02aab87b9a P 782a2eeace8d4dc68f55d03c1c745d3c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 07:58:12.768767 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "229fec0abb91487582fad4da9f264ef5" candidate_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615b7a2a3e424b8ca8b1182212a0e945" is_pre_election: true
I20240502 07:58:12.769055 30793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321c5b918bc14901908bf94c28e0d4e3" candidate_uuid: "c9ffe0c529954a0f88581546fc6573d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" is_pre_election: true
W20240502 07:58:12.769165 30758 leader_election.cc:341] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 615b7a2a3e424b8ca8b1182212a0e945 (127.29.106.130:34139): Illegal state: must be running to vote when last-logged opid is not known
I20240502 07:58:12.769266 30793 raft_consensus.cc:2454] T 321c5b918bc14901908bf94c28e0d4e3 P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9ffe0c529954a0f88581546fc6573d3 in term 0.
I20240502 07:58:12.769317 30991 raft_consensus.cc:348] T d37ebaee499a4240a6ecfb02aab87b9a P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.769407 30991 raft_consensus.cc:374] T d37ebaee499a4240a6ecfb02aab87b9a P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:12.769438 30991 raft_consensus.cc:729] T d37ebaee499a4240a6ecfb02aab87b9a P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 782a2eeace8d4dc68f55d03c1c745d3c, State: Initialized, Role: FOLLOWER
I20240502 07:58:12.769474 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321c5b918bc14901908bf94c28e0d4e3" candidate_uuid: "c9ffe0c529954a0f88581546fc6573d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615b7a2a3e424b8ca8b1182212a0e945" is_pre_election: true
I20240502 07:58:12.769584 30870 raft_consensus.cc:2454] T 321c5b918bc14901908bf94c28e0d4e3 P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9ffe0c529954a0f88581546fc6573d3 in term 0.
I20240502 07:58:12.769590 30991 consensus_queue.cc:260] T d37ebaee499a4240a6ecfb02aab87b9a P 782a2eeace8d4dc68f55d03c1c745d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.769858 30909 leader_election.cc:302] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615b7a2a3e424b8ca8b1182212a0e945, c9ffe0c529954a0f88581546fc6573d3; no voters:
I20240502 07:58:12.770107 30991 ts_tablet_manager.cc:1398] T d37ebaee499a4240a6ecfb02aab87b9a P 782a2eeace8d4dc68f55d03c1c745d3c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:12.770370 30991 tablet_bootstrap.cc:492] T 3175330f9b374e559549707d1107306b P 782a2eeace8d4dc68f55d03c1c745d3c: Bootstrap starting.
I20240502 07:58:12.770442 31022 raft_consensus.cc:2781] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:12.770449 30945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "229fec0abb91487582fad4da9f264ef5" candidate_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9ffe0c529954a0f88581546fc6573d3" is_pre_election: true
I20240502 07:58:12.770524 31022 raft_consensus.cc:482] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:12.770706 31022 raft_consensus.cc:3037] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:12.771165 30992 tablet_bootstrap.cc:492] T d37ebaee499a4240a6ecfb02aab87b9a P 615b7a2a3e424b8ca8b1182212a0e945: No bootstrap required, opened a new log
I20240502 07:58:12.771276 30992 ts_tablet_manager.cc:1367] T d37ebaee499a4240a6ecfb02aab87b9a P 615b7a2a3e424b8ca8b1182212a0e945: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 07:58:12.771764 30991 tablet_bootstrap.cc:654] T 3175330f9b374e559549707d1107306b P 782a2eeace8d4dc68f55d03c1c745d3c: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:12.771752 30992 raft_consensus.cc:348] T d37ebaee499a4240a6ecfb02aab87b9a P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.771880 30992 raft_consensus.cc:374] T d37ebaee499a4240a6ecfb02aab87b9a P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:12.771935 30992 raft_consensus.cc:729] T d37ebaee499a4240a6ecfb02aab87b9a P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615b7a2a3e424b8ca8b1182212a0e945, State: Initialized, Role: FOLLOWER
I20240502 07:58:12.772055 30992 consensus_queue.cc:260] T d37ebaee499a4240a6ecfb02aab87b9a P 615b7a2a3e424b8ca8b1182212a0e945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.772365 30992 ts_tablet_manager.cc:1398] T d37ebaee499a4240a6ecfb02aab87b9a P 615b7a2a3e424b8ca8b1182212a0e945: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:12.772588 30992 tablet_bootstrap.cc:492] T 229fec0abb91487582fad4da9f264ef5 P 615b7a2a3e424b8ca8b1182212a0e945: Bootstrap starting.
I20240502 07:58:12.773801 30991 tablet_bootstrap.cc:492] T 3175330f9b374e559549707d1107306b P 782a2eeace8d4dc68f55d03c1c745d3c: No bootstrap required, opened a new log
I20240502 07:58:12.773865 30992 tablet_bootstrap.cc:654] T 229fec0abb91487582fad4da9f264ef5 P 615b7a2a3e424b8ca8b1182212a0e945: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:12.773905 30991 ts_tablet_manager.cc:1367] T 3175330f9b374e559549707d1107306b P 782a2eeace8d4dc68f55d03c1c745d3c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:58:12.774415 30991 raft_consensus.cc:348] T 3175330f9b374e559549707d1107306b P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.774551 30991 raft_consensus.cc:374] T 3175330f9b374e559549707d1107306b P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:12.774587 30991 raft_consensus.cc:729] T 3175330f9b374e559549707d1107306b P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 782a2eeace8d4dc68f55d03c1c745d3c, State: Initialized, Role: FOLLOWER
I20240502 07:58:12.774722 30991 consensus_queue.cc:260] T 3175330f9b374e559549707d1107306b P 782a2eeace8d4dc68f55d03c1c745d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.775101 30991 ts_tablet_manager.cc:1398] T 3175330f9b374e559549707d1107306b P 782a2eeace8d4dc68f55d03c1c745d3c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:12.775741 30992 tablet_bootstrap.cc:492] T 229fec0abb91487582fad4da9f264ef5 P 615b7a2a3e424b8ca8b1182212a0e945: No bootstrap required, opened a new log
I20240502 07:58:12.775838 30992 ts_tablet_manager.cc:1367] T 229fec0abb91487582fad4da9f264ef5 P 615b7a2a3e424b8ca8b1182212a0e945: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:12.776305 30992 raft_consensus.cc:348] T 229fec0abb91487582fad4da9f264ef5 P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.776448 30992 raft_consensus.cc:374] T 229fec0abb91487582fad4da9f264ef5 P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:12.776504 30992 raft_consensus.cc:729] T 229fec0abb91487582fad4da9f264ef5 P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615b7a2a3e424b8ca8b1182212a0e945, State: Initialized, Role: FOLLOWER
I20240502 07:58:12.776640 30992 consensus_queue.cc:260] T 229fec0abb91487582fad4da9f264ef5 P 615b7a2a3e424b8ca8b1182212a0e945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.776974 30992 ts_tablet_manager.cc:1398] T 229fec0abb91487582fad4da9f264ef5 P 615b7a2a3e424b8ca8b1182212a0e945: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:12.777324 30992 tablet_bootstrap.cc:492] T 46c0c7f5bb734a19a515bf0869de62d6 P 615b7a2a3e424b8ca8b1182212a0e945: Bootstrap starting.
I20240502 07:58:12.778398 30992 tablet_bootstrap.cc:654] T 46c0c7f5bb734a19a515bf0869de62d6 P 615b7a2a3e424b8ca8b1182212a0e945: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:12.780072 30992 tablet_bootstrap.cc:492] T 46c0c7f5bb734a19a515bf0869de62d6 P 615b7a2a3e424b8ca8b1182212a0e945: No bootstrap required, opened a new log
I20240502 07:58:12.780186 30992 ts_tablet_manager.cc:1367] T 46c0c7f5bb734a19a515bf0869de62d6 P 615b7a2a3e424b8ca8b1182212a0e945: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:12.780623 30992 raft_consensus.cc:348] T 46c0c7f5bb734a19a515bf0869de62d6 P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.780741 30992 raft_consensus.cc:374] T 46c0c7f5bb734a19a515bf0869de62d6 P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:12.780766 30992 raft_consensus.cc:729] T 46c0c7f5bb734a19a515bf0869de62d6 P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615b7a2a3e424b8ca8b1182212a0e945, State: Initialized, Role: FOLLOWER
I20240502 07:58:12.780880 30992 consensus_queue.cc:260] T 46c0c7f5bb734a19a515bf0869de62d6 P 615b7a2a3e424b8ca8b1182212a0e945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.781194 30992 ts_tablet_manager.cc:1398] T 46c0c7f5bb734a19a515bf0869de62d6 P 615b7a2a3e424b8ca8b1182212a0e945: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:12.781445 30992 tablet_bootstrap.cc:492] T 98055797532e45b3a72d6ef4f278bd76 P 615b7a2a3e424b8ca8b1182212a0e945: Bootstrap starting.
I20240502 07:58:12.782516 30992 tablet_bootstrap.cc:654] T 98055797532e45b3a72d6ef4f278bd76 P 615b7a2a3e424b8ca8b1182212a0e945: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:12.783807 30992 tablet_bootstrap.cc:492] T 98055797532e45b3a72d6ef4f278bd76 P 615b7a2a3e424b8ca8b1182212a0e945: No bootstrap required, opened a new log
I20240502 07:58:12.783896 30992 ts_tablet_manager.cc:1367] T 98055797532e45b3a72d6ef4f278bd76 P 615b7a2a3e424b8ca8b1182212a0e945: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:12.784350 30992 raft_consensus.cc:348] T 98055797532e45b3a72d6ef4f278bd76 P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.784451 30992 raft_consensus.cc:374] T 98055797532e45b3a72d6ef4f278bd76 P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:12.784474 30992 raft_consensus.cc:729] T 98055797532e45b3a72d6ef4f278bd76 P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615b7a2a3e424b8ca8b1182212a0e945, State: Initialized, Role: FOLLOWER
I20240502 07:58:12.784574 30992 consensus_queue.cc:260] T 98055797532e45b3a72d6ef4f278bd76 P 615b7a2a3e424b8ca8b1182212a0e945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.784837 30992 ts_tablet_manager.cc:1398] T 98055797532e45b3a72d6ef4f278bd76 P 615b7a2a3e424b8ca8b1182212a0e945: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:12.802495 31002 raft_consensus.cc:482] T 3175330f9b374e559549707d1107306b P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:12.802744 31002 raft_consensus.cc:504] T 3175330f9b374e559549707d1107306b P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.803292 31002 leader_election.cc:288] T 3175330f9b374e559549707d1107306b P 782a2eeace8d4dc68f55d03c1c745d3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 615b7a2a3e424b8ca8b1182212a0e945 (127.29.106.130:34139), c9ffe0c529954a0f88581546fc6573d3 (127.29.106.131:34467)
I20240502 07:58:12.803879 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3175330f9b374e559549707d1107306b" candidate_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615b7a2a3e424b8ca8b1182212a0e945" is_pre_election: true
I20240502 07:58:12.803897 30944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3175330f9b374e559549707d1107306b" candidate_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9ffe0c529954a0f88581546fc6573d3" is_pre_election: true
I20240502 07:58:12.804100 30870 raft_consensus.cc:2454] T 3175330f9b374e559549707d1107306b P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 782a2eeace8d4dc68f55d03c1c745d3c in term 0.
I20240502 07:58:12.804488 30758 leader_election.cc:302] T 3175330f9b374e559549707d1107306b P 782a2eeace8d4dc68f55d03c1c745d3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615b7a2a3e424b8ca8b1182212a0e945, 782a2eeace8d4dc68f55d03c1c745d3c; no voters:
I20240502 07:58:12.804734 31002 raft_consensus.cc:2781] T 3175330f9b374e559549707d1107306b P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:12.804805 31002 raft_consensus.cc:482] T 3175330f9b374e559549707d1107306b P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:12.804836 31002 raft_consensus.cc:3037] T 3175330f9b374e559549707d1107306b P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Advancing to term 1
W20240502 07:58:12.835179 30821 tablet.cc:2369] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:58:12.843927 31016 raft_consensus.cc:482] T 46c0c7f5bb734a19a515bf0869de62d6 P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:12.844139 31016 raft_consensus.cc:504] T 46c0c7f5bb734a19a515bf0869de62d6 P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.844704 31016 leader_election.cc:288] T 46c0c7f5bb734a19a515bf0869de62d6 P 782a2eeace8d4dc68f55d03c1c745d3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 615b7a2a3e424b8ca8b1182212a0e945 (127.29.106.130:34139), c9ffe0c529954a0f88581546fc6573d3 (127.29.106.131:34467)
I20240502 07:58:12.845288 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46c0c7f5bb734a19a515bf0869de62d6" candidate_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615b7a2a3e424b8ca8b1182212a0e945" is_pre_election: true
I20240502 07:58:12.845418 30870 raft_consensus.cc:2454] T 46c0c7f5bb734a19a515bf0869de62d6 P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 782a2eeace8d4dc68f55d03c1c745d3c in term 0.
I20240502 07:58:12.845741 30758 leader_election.cc:302] T 46c0c7f5bb734a19a515bf0869de62d6 P 782a2eeace8d4dc68f55d03c1c745d3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615b7a2a3e424b8ca8b1182212a0e945, 782a2eeace8d4dc68f55d03c1c745d3c; no voters:
I20240502 07:58:12.845700 30943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46c0c7f5bb734a19a515bf0869de62d6" candidate_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9ffe0c529954a0f88581546fc6573d3" is_pre_election: true
I20240502 07:58:12.845974 31016 raft_consensus.cc:2781] T 46c0c7f5bb734a19a515bf0869de62d6 P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:12.845927 30943 raft_consensus.cc:2454] T 46c0c7f5bb734a19a515bf0869de62d6 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 782a2eeace8d4dc68f55d03c1c745d3c in term 0.
I20240502 07:58:12.846037 31016 raft_consensus.cc:482] T 46c0c7f5bb734a19a515bf0869de62d6 P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:12.846076 31016 raft_consensus.cc:3037] T 46c0c7f5bb734a19a515bf0869de62d6 P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:12.851630 31002 raft_consensus.cc:504] T 3175330f9b374e559549707d1107306b P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.852697 31020 raft_consensus.cc:504] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.852882 30943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3175330f9b374e559549707d1107306b" candidate_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9ffe0c529954a0f88581546fc6573d3"
I20240502 07:58:12.853039 31020 leader_election.cc:288] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [CANDIDATE]: Term 1 election: Requested vote from peers 782a2eeace8d4dc68f55d03c1c745d3c (127.29.106.129:41727), 615b7a2a3e424b8ca8b1182212a0e945 (127.29.106.130:34139)
I20240502 07:58:12.853200 30945 raft_consensus.cc:2379] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 782a2eeace8d4dc68f55d03c1c745d3c in current term 1: Already voted for candidate c9ffe0c529954a0f88581546fc6573d3 in this term.
I20240502 07:58:12.851631 30996 raft_consensus.cc:504] T 3175330f9b374e559549707d1107306b P c9ffe0c529954a0f88581546fc6573d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.853088 31022 raft_consensus.cc:504] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.853726 30757 leader_election.cc:302] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 782a2eeace8d4dc68f55d03c1c745d3c; no voters: 615b7a2a3e424b8ca8b1182212a0e945, c9ffe0c529954a0f88581546fc6573d3
I20240502 07:58:12.853756 30996 leader_election.cc:288] T 3175330f9b374e559549707d1107306b P c9ffe0c529954a0f88581546fc6573d3 [CANDIDATE]: Term 1 election: Requested vote from peers 782a2eeace8d4dc68f55d03c1c745d3c (127.29.106.129:41727), 615b7a2a3e424b8ca8b1182212a0e945 (127.29.106.130:34139)
I20240502 07:58:12.853802 31022 leader_election.cc:288] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3 [CANDIDATE]: Term 1 election: Requested vote from peers 782a2eeace8d4dc68f55d03c1c745d3c (127.29.106.129:41727), 615b7a2a3e424b8ca8b1182212a0e945 (127.29.106.130:34139)
I20240502 07:58:12.853893 31002 leader_election.cc:288] T 3175330f9b374e559549707d1107306b P 782a2eeace8d4dc68f55d03c1c745d3c [CANDIDATE]: Term 1 election: Requested vote from peers 615b7a2a3e424b8ca8b1182212a0e945 (127.29.106.130:34139), c9ffe0c529954a0f88581546fc6573d3 (127.29.106.131:34467)
I20240502 07:58:12.853998 31002 raft_consensus.cc:3037] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:12.854166 30944 raft_consensus.cc:2379] T 3175330f9b374e559549707d1107306b P c9ffe0c529954a0f88581546fc6573d3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 782a2eeace8d4dc68f55d03c1c745d3c in current term 1: Already voted for candidate c9ffe0c529954a0f88581546fc6573d3 in this term.
I20240502 07:58:12.854399 30793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "229fec0abb91487582fad4da9f264ef5" candidate_uuid: "c9ffe0c529954a0f88581546fc6573d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782a2eeace8d4dc68f55d03c1c745d3c"
I20240502 07:58:12.854619 30794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3175330f9b374e559549707d1107306b" candidate_uuid: "c9ffe0c529954a0f88581546fc6573d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782a2eeace8d4dc68f55d03c1c745d3c"
I20240502 07:58:12.854725 30794 raft_consensus.cc:2379] T 3175330f9b374e559549707d1107306b P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c9ffe0c529954a0f88581546fc6573d3 in current term 1: Already voted for candidate 782a2eeace8d4dc68f55d03c1c745d3c in this term.
I20240502 07:58:12.854923 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321c5b918bc14901908bf94c28e0d4e3" candidate_uuid: "c9ffe0c529954a0f88581546fc6573d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782a2eeace8d4dc68f55d03c1c745d3c"
I20240502 07:58:12.855005 31002 raft_consensus.cc:2726] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 07:58:12.855022 30792 raft_consensus.cc:3037] T 321c5b918bc14901908bf94c28e0d4e3 P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:12.855115 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321c5b918bc14901908bf94c28e0d4e3" candidate_uuid: "c9ffe0c529954a0f88581546fc6573d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615b7a2a3e424b8ca8b1182212a0e945"
I20240502 07:58:12.855253 30868 raft_consensus.cc:3037] T 321c5b918bc14901908bf94c28e0d4e3 P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:12.855020 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3175330f9b374e559549707d1107306b" candidate_uuid: "c9ffe0c529954a0f88581546fc6573d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615b7a2a3e424b8ca8b1182212a0e945"
I20240502 07:58:12.855880 30869 raft_consensus.cc:3037] T 3175330f9b374e559549707d1107306b P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:12.856041 30792 raft_consensus.cc:2454] T 321c5b918bc14901908bf94c28e0d4e3 P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9ffe0c529954a0f88581546fc6573d3 in term 1.
I20240502 07:58:12.856679 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "229fec0abb91487582fad4da9f264ef5" candidate_uuid: "c9ffe0c529954a0f88581546fc6573d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615b7a2a3e424b8ca8b1182212a0e945"
I20240502 07:58:12.856808 30870 raft_consensus.cc:3037] T 229fec0abb91487582fad4da9f264ef5 P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:12.856842 30911 leader_election.cc:302] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 782a2eeace8d4dc68f55d03c1c745d3c, c9ffe0c529954a0f88581546fc6573d3; no voters:
I20240502 07:58:12.857012 31022 raft_consensus.cc:2781] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:12.857089 31022 raft_consensus.cc:686] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3 [term 1 LEADER]: Becoming Leader. State: Replica: c9ffe0c529954a0f88581546fc6573d3, State: Running, Role: LEADER
I20240502 07:58:12.857241 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3175330f9b374e559549707d1107306b" candidate_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615b7a2a3e424b8ca8b1182212a0e945"
I20240502 07:58:12.857245 31022 consensus_queue.cc:237] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.857630 30758 leader_election.cc:302] T 3175330f9b374e559549707d1107306b P 782a2eeace8d4dc68f55d03c1c745d3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 782a2eeace8d4dc68f55d03c1c745d3c; no voters: 615b7a2a3e424b8ca8b1182212a0e945, c9ffe0c529954a0f88581546fc6573d3
I20240502 07:58:12.857803 31029 raft_consensus.cc:2726] T 3175330f9b374e559549707d1107306b P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 07:58:12.859220 30997 raft_consensus.cc:482] T e6509f5318594a9d8e8d8833202652e7 P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:12.859337 30997 raft_consensus.cc:504] T e6509f5318594a9d8e8d8833202652e7 P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.859280 30690 catalog_manager.cc:5484] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9ffe0c529954a0f88581546fc6573d3 (127.29.106.131). New cstate: current_term: 1 leader_uuid: "c9ffe0c529954a0f88581546fc6573d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:12.859884 30997 leader_election.cc:288] T e6509f5318594a9d8e8d8833202652e7 P 615b7a2a3e424b8ca8b1182212a0e945 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 782a2eeace8d4dc68f55d03c1c745d3c (127.29.106.129:41727), c9ffe0c529954a0f88581546fc6573d3 (127.29.106.131:34467)
I20240502 07:58:12.865059 30944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6509f5318594a9d8e8d8833202652e7" candidate_uuid: "615b7a2a3e424b8ca8b1182212a0e945" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9ffe0c529954a0f88581546fc6573d3" is_pre_election: true
I20240502 07:58:12.865288 30944 raft_consensus.cc:2454] T e6509f5318594a9d8e8d8833202652e7 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 615b7a2a3e424b8ca8b1182212a0e945 in term 0.
I20240502 07:58:12.865437 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6509f5318594a9d8e8d8833202652e7" candidate_uuid: "615b7a2a3e424b8ca8b1182212a0e945" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" is_pre_election: true
I20240502 07:58:12.865541 30792 raft_consensus.cc:2454] T e6509f5318594a9d8e8d8833202652e7 P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 615b7a2a3e424b8ca8b1182212a0e945 in term 0.
I20240502 07:58:12.865633 30833 leader_election.cc:302] T e6509f5318594a9d8e8d8833202652e7 P 615b7a2a3e424b8ca8b1182212a0e945 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615b7a2a3e424b8ca8b1182212a0e945, c9ffe0c529954a0f88581546fc6573d3; no voters:
I20240502 07:58:12.865839 30997 raft_consensus.cc:2781] T e6509f5318594a9d8e8d8833202652e7 P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:12.865943 30997 raft_consensus.cc:482] T e6509f5318594a9d8e8d8833202652e7 P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:12.866029 30997 raft_consensus.cc:3037] T e6509f5318594a9d8e8d8833202652e7 P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:12.913262 30793 raft_consensus.cc:2454] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9ffe0c529954a0f88581546fc6573d3 in term 1.
I20240502 07:58:12.913851 30911 leader_election.cc:302] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 782a2eeace8d4dc68f55d03c1c745d3c, c9ffe0c529954a0f88581546fc6573d3; no voters:
I20240502 07:58:12.914115 31022 raft_consensus.cc:2781] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:12.914288 31022 raft_consensus.cc:686] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [term 1 LEADER]: Becoming Leader. State: Replica: c9ffe0c529954a0f88581546fc6573d3, State: Running, Role: LEADER
I20240502 07:58:12.914513 31022 consensus_queue.cc:237] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.916882 30690 catalog_manager.cc:5484] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9ffe0c529954a0f88581546fc6573d3 (127.29.106.131). New cstate: current_term: 1 leader_uuid: "c9ffe0c529954a0f88581546fc6573d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:12.945513 30868 raft_consensus.cc:2454] T 321c5b918bc14901908bf94c28e0d4e3 P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9ffe0c529954a0f88581546fc6573d3 in term 1.
I20240502 07:58:12.945758 30869 raft_consensus.cc:2454] T 3175330f9b374e559549707d1107306b P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9ffe0c529954a0f88581546fc6573d3 in term 1.
I20240502 07:58:12.945508 31016 raft_consensus.cc:504] T 46c0c7f5bb734a19a515bf0869de62d6 P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.946491 30909 leader_election.cc:302] T 3175330f9b374e559549707d1107306b P c9ffe0c529954a0f88581546fc6573d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 615b7a2a3e424b8ca8b1182212a0e945, c9ffe0c529954a0f88581546fc6573d3; no voters: 782a2eeace8d4dc68f55d03c1c745d3c
I20240502 07:58:12.946594 31016 leader_election.cc:288] T 46c0c7f5bb734a19a515bf0869de62d6 P 782a2eeace8d4dc68f55d03c1c745d3c [CANDIDATE]: Term 1 election: Requested vote from peers 615b7a2a3e424b8ca8b1182212a0e945 (127.29.106.130:34139), c9ffe0c529954a0f88581546fc6573d3 (127.29.106.131:34467)
I20240502 07:58:12.946739 31022 raft_consensus.cc:2781] T 3175330f9b374e559549707d1107306b P c9ffe0c529954a0f88581546fc6573d3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:12.946815 31022 raft_consensus.cc:686] T 3175330f9b374e559549707d1107306b P c9ffe0c529954a0f88581546fc6573d3 [term 1 LEADER]: Becoming Leader. State: Replica: c9ffe0c529954a0f88581546fc6573d3, State: Running, Role: LEADER
I20240502 07:58:12.947000 30944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46c0c7f5bb734a19a515bf0869de62d6" candidate_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9ffe0c529954a0f88581546fc6573d3"
I20240502 07:58:12.946979 31022 consensus_queue.cc:237] T 3175330f9b374e559549707d1107306b P c9ffe0c529954a0f88581546fc6573d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.947130 30944 raft_consensus.cc:3037] T 46c0c7f5bb734a19a515bf0869de62d6 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:12.947274 30870 raft_consensus.cc:2454] T 229fec0abb91487582fad4da9f264ef5 P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9ffe0c529954a0f88581546fc6573d3 in term 1.
I20240502 07:58:12.947336 30997 raft_consensus.cc:504] T e6509f5318594a9d8e8d8833202652e7 P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.947796 30997 leader_election.cc:288] T e6509f5318594a9d8e8d8833202652e7 P 615b7a2a3e424b8ca8b1182212a0e945 [CANDIDATE]: Term 1 election: Requested vote from peers 782a2eeace8d4dc68f55d03c1c745d3c (127.29.106.129:41727), c9ffe0c529954a0f88581546fc6573d3 (127.29.106.131:34467)
I20240502 07:58:12.948316 30945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6509f5318594a9d8e8d8833202652e7" candidate_uuid: "615b7a2a3e424b8ca8b1182212a0e945" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9ffe0c529954a0f88581546fc6573d3"
I20240502 07:58:12.948323 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46c0c7f5bb734a19a515bf0869de62d6" candidate_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615b7a2a3e424b8ca8b1182212a0e945"
I20240502 07:58:12.948459 30945 raft_consensus.cc:3037] T e6509f5318594a9d8e8d8833202652e7 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:12.948487 30870 raft_consensus.cc:3037] T 46c0c7f5bb734a19a515bf0869de62d6 P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:12.948539 30793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6509f5318594a9d8e8d8833202652e7" candidate_uuid: "615b7a2a3e424b8ca8b1182212a0e945" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782a2eeace8d4dc68f55d03c1c745d3c"
I20240502 07:58:12.948617 30793 raft_consensus.cc:3037] T e6509f5318594a9d8e8d8833202652e7 P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:12.949137 30690 catalog_manager.cc:5484] T 3175330f9b374e559549707d1107306b P c9ffe0c529954a0f88581546fc6573d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9ffe0c529954a0f88581546fc6573d3 (127.29.106.131). New cstate: current_term: 1 leader_uuid: "c9ffe0c529954a0f88581546fc6573d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:12.949533 30870 raft_consensus.cc:2454] T 46c0c7f5bb734a19a515bf0869de62d6 P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 782a2eeace8d4dc68f55d03c1c745d3c in term 1.
I20240502 07:58:12.949533 30793 raft_consensus.cc:2454] T e6509f5318594a9d8e8d8833202652e7 P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615b7a2a3e424b8ca8b1182212a0e945 in term 1.
I20240502 07:58:12.949915 30758 leader_election.cc:302] T 46c0c7f5bb734a19a515bf0869de62d6 P 782a2eeace8d4dc68f55d03c1c745d3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615b7a2a3e424b8ca8b1182212a0e945, 782a2eeace8d4dc68f55d03c1c745d3c; no voters:
I20240502 07:58:12.950023 30836 leader_election.cc:302] T e6509f5318594a9d8e8d8833202652e7 P 615b7a2a3e424b8ca8b1182212a0e945 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615b7a2a3e424b8ca8b1182212a0e945, 782a2eeace8d4dc68f55d03c1c745d3c; no voters:
I20240502 07:58:12.950086 31016 raft_consensus.cc:2781] T 46c0c7f5bb734a19a515bf0869de62d6 P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:12.950229 30997 raft_consensus.cc:2781] T e6509f5318594a9d8e8d8833202652e7 P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:12.950464 30997 raft_consensus.cc:686] T e6509f5318594a9d8e8d8833202652e7 P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 LEADER]: Becoming Leader. State: Replica: 615b7a2a3e424b8ca8b1182212a0e945, State: Running, Role: LEADER
I20240502 07:58:12.950608 31016 raft_consensus.cc:686] T 46c0c7f5bb734a19a515bf0869de62d6 P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 LEADER]: Becoming Leader. State: Replica: 782a2eeace8d4dc68f55d03c1c745d3c, State: Running, Role: LEADER
I20240502 07:58:12.950721 30997 consensus_queue.cc:237] T e6509f5318594a9d8e8d8833202652e7 P 615b7a2a3e424b8ca8b1182212a0e945 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.950989 31016 consensus_queue.cc:237] T 46c0c7f5bb734a19a515bf0869de62d6 P 782a2eeace8d4dc68f55d03c1c745d3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.952708 30690 catalog_manager.cc:5484] T e6509f5318594a9d8e8d8833202652e7 P 615b7a2a3e424b8ca8b1182212a0e945 reported cstate change: term changed from 0 to 1, leader changed from <none> to 615b7a2a3e424b8ca8b1182212a0e945 (127.29.106.130). New cstate: current_term: 1 leader_uuid: "615b7a2a3e424b8ca8b1182212a0e945" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:12.953157 30691 catalog_manager.cc:5484] T 46c0c7f5bb734a19a515bf0869de62d6 P 782a2eeace8d4dc68f55d03c1c745d3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 782a2eeace8d4dc68f55d03c1c745d3c (127.29.106.129). New cstate: current_term: 1 leader_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:12.958040 31022 raft_consensus.cc:482] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:12.958161 31022 raft_consensus.cc:504] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.958562 31022 leader_election.cc:288] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 782a2eeace8d4dc68f55d03c1c745d3c (127.29.106.129:41727), 615b7a2a3e424b8ca8b1182212a0e945 (127.29.106.130:34139)
I20240502 07:58:12.958979 30793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98055797532e45b3a72d6ef4f278bd76" candidate_uuid: "c9ffe0c529954a0f88581546fc6573d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" is_pre_election: true
I20240502 07:58:12.958984 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98055797532e45b3a72d6ef4f278bd76" candidate_uuid: "c9ffe0c529954a0f88581546fc6573d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615b7a2a3e424b8ca8b1182212a0e945" is_pre_election: true
I20240502 07:58:12.959170 30793 raft_consensus.cc:2454] T 98055797532e45b3a72d6ef4f278bd76 P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9ffe0c529954a0f88581546fc6573d3 in term 0.
I20240502 07:58:12.959283 30870 raft_consensus.cc:2454] T 98055797532e45b3a72d6ef4f278bd76 P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9ffe0c529954a0f88581546fc6573d3 in term 0.
I20240502 07:58:12.959547 30909 leader_election.cc:302] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615b7a2a3e424b8ca8b1182212a0e945, c9ffe0c529954a0f88581546fc6573d3; no voters:
I20240502 07:58:12.959702 31022 raft_consensus.cc:2781] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:12.959764 31022 raft_consensus.cc:482] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:12.959828 31022 raft_consensus.cc:3037] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:12.977980 30945 raft_consensus.cc:2454] T e6509f5318594a9d8e8d8833202652e7 P c9ffe0c529954a0f88581546fc6573d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615b7a2a3e424b8ca8b1182212a0e945 in term 1.
I20240502 07:58:12.978034 31022 raft_consensus.cc:504] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.978201 30944 raft_consensus.cc:2454] T 46c0c7f5bb734a19a515bf0869de62d6 P c9ffe0c529954a0f88581546fc6573d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 782a2eeace8d4dc68f55d03c1c745d3c in term 1.
I20240502 07:58:12.978475 31022 leader_election.cc:288] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3 [CANDIDATE]: Term 1 election: Requested vote from peers 782a2eeace8d4dc68f55d03c1c745d3c (127.29.106.129:41727), 615b7a2a3e424b8ca8b1182212a0e945 (127.29.106.130:34139)
I20240502 07:58:12.979101 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98055797532e45b3a72d6ef4f278bd76" candidate_uuid: "c9ffe0c529954a0f88581546fc6573d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615b7a2a3e424b8ca8b1182212a0e945"
I20240502 07:58:12.979178 30793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98055797532e45b3a72d6ef4f278bd76" candidate_uuid: "c9ffe0c529954a0f88581546fc6573d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782a2eeace8d4dc68f55d03c1c745d3c"
I20240502 07:58:12.979225 30870 raft_consensus.cc:3037] T 98055797532e45b3a72d6ef4f278bd76 P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:12.979246 30793 raft_consensus.cc:3037] T 98055797532e45b3a72d6ef4f278bd76 P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:12.980196 30793 raft_consensus.cc:2454] T 98055797532e45b3a72d6ef4f278bd76 P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9ffe0c529954a0f88581546fc6573d3 in term 1.
I20240502 07:58:12.980196 30870 raft_consensus.cc:2454] T 98055797532e45b3a72d6ef4f278bd76 P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9ffe0c529954a0f88581546fc6573d3 in term 1.
I20240502 07:58:12.980688 30911 leader_election.cc:302] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 782a2eeace8d4dc68f55d03c1c745d3c, c9ffe0c529954a0f88581546fc6573d3; no voters:
I20240502 07:58:12.980950 31022 raft_consensus.cc:2781] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:12.981058 31022 raft_consensus.cc:686] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3 [term 1 LEADER]: Becoming Leader. State: Replica: c9ffe0c529954a0f88581546fc6573d3, State: Running, Role: LEADER
I20240502 07:58:12.981281 31022 consensus_queue.cc:237] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:12.983016 30691 catalog_manager.cc:5484] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9ffe0c529954a0f88581546fc6573d3 (127.29.106.131). New cstate: current_term: 1 leader_uuid: "c9ffe0c529954a0f88581546fc6573d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:13.003222 30997 raft_consensus.cc:482] T d37ebaee499a4240a6ecfb02aab87b9a P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:13.003401 30997 raft_consensus.cc:504] T d37ebaee499a4240a6ecfb02aab87b9a P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:13.003893 30997 leader_election.cc:288] T d37ebaee499a4240a6ecfb02aab87b9a P 615b7a2a3e424b8ca8b1182212a0e945 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 782a2eeace8d4dc68f55d03c1c745d3c (127.29.106.129:41727), c9ffe0c529954a0f88581546fc6573d3 (127.29.106.131:34467)
I20240502 07:58:13.004711 30793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37ebaee499a4240a6ecfb02aab87b9a" candidate_uuid: "615b7a2a3e424b8ca8b1182212a0e945" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" is_pre_election: true
I20240502 07:58:13.004735 30944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37ebaee499a4240a6ecfb02aab87b9a" candidate_uuid: "615b7a2a3e424b8ca8b1182212a0e945" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9ffe0c529954a0f88581546fc6573d3" is_pre_election: true
I20240502 07:58:13.004837 30793 raft_consensus.cc:2454] T d37ebaee499a4240a6ecfb02aab87b9a P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 615b7a2a3e424b8ca8b1182212a0e945 in term 0.
I20240502 07:58:13.004995 30944 raft_consensus.cc:2454] T d37ebaee499a4240a6ecfb02aab87b9a P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 615b7a2a3e424b8ca8b1182212a0e945 in term 0.
I20240502 07:58:13.005358 30833 leader_election.cc:302] T d37ebaee499a4240a6ecfb02aab87b9a P 615b7a2a3e424b8ca8b1182212a0e945 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615b7a2a3e424b8ca8b1182212a0e945, c9ffe0c529954a0f88581546fc6573d3; no voters:
I20240502 07:58:13.005574 30997 raft_consensus.cc:2781] T d37ebaee499a4240a6ecfb02aab87b9a P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:13.005658 30997 raft_consensus.cc:482] T d37ebaee499a4240a6ecfb02aab87b9a P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:13.005733 30997 raft_consensus.cc:3037] T d37ebaee499a4240a6ecfb02aab87b9a P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:13.073858 30997 raft_consensus.cc:504] T d37ebaee499a4240a6ecfb02aab87b9a P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:13.074450 30997 leader_election.cc:288] T d37ebaee499a4240a6ecfb02aab87b9a P 615b7a2a3e424b8ca8b1182212a0e945 [CANDIDATE]: Term 1 election: Requested vote from peers 782a2eeace8d4dc68f55d03c1c745d3c (127.29.106.129:41727), c9ffe0c529954a0f88581546fc6573d3 (127.29.106.131:34467)
I20240502 07:58:13.075109 30793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37ebaee499a4240a6ecfb02aab87b9a" candidate_uuid: "615b7a2a3e424b8ca8b1182212a0e945" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782a2eeace8d4dc68f55d03c1c745d3c"
I20240502 07:58:13.075109 30944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37ebaee499a4240a6ecfb02aab87b9a" candidate_uuid: "615b7a2a3e424b8ca8b1182212a0e945" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9ffe0c529954a0f88581546fc6573d3"
I20240502 07:58:13.075280 30793 raft_consensus.cc:3037] T d37ebaee499a4240a6ecfb02aab87b9a P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:13.075290 30944 raft_consensus.cc:3037] T d37ebaee499a4240a6ecfb02aab87b9a P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:13.076364 30793 raft_consensus.cc:2454] T d37ebaee499a4240a6ecfb02aab87b9a P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615b7a2a3e424b8ca8b1182212a0e945 in term 1.
I20240502 07:58:13.076364 30944 raft_consensus.cc:2454] T d37ebaee499a4240a6ecfb02aab87b9a P c9ffe0c529954a0f88581546fc6573d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615b7a2a3e424b8ca8b1182212a0e945 in term 1.
I20240502 07:58:13.076743 30833 leader_election.cc:302] T d37ebaee499a4240a6ecfb02aab87b9a P 615b7a2a3e424b8ca8b1182212a0e945 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615b7a2a3e424b8ca8b1182212a0e945, c9ffe0c529954a0f88581546fc6573d3; no voters:
I20240502 07:58:13.076979 30997 raft_consensus.cc:2781] T d37ebaee499a4240a6ecfb02aab87b9a P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:13.077075 30997 raft_consensus.cc:686] T d37ebaee499a4240a6ecfb02aab87b9a P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 LEADER]: Becoming Leader. State: Replica: 615b7a2a3e424b8ca8b1182212a0e945, State: Running, Role: LEADER
I20240502 07:58:13.077216 30997 consensus_queue.cc:237] T d37ebaee499a4240a6ecfb02aab87b9a P 615b7a2a3e424b8ca8b1182212a0e945 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:13.078917 30691 catalog_manager.cc:5484] T d37ebaee499a4240a6ecfb02aab87b9a P 615b7a2a3e424b8ca8b1182212a0e945 reported cstate change: term changed from 0 to 1, leader changed from <none> to 615b7a2a3e424b8ca8b1182212a0e945 (127.29.106.130). New cstate: current_term: 1 leader_uuid: "615b7a2a3e424b8ca8b1182212a0e945" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:13.148785 30997 raft_consensus.cc:482] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:13.148941 30997 raft_consensus.cc:504] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:13.149369 30997 leader_election.cc:288] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 782a2eeace8d4dc68f55d03c1c745d3c (127.29.106.129:41727), c9ffe0c529954a0f88581546fc6573d3 (127.29.106.131:34467)
I20240502 07:58:13.150130 30944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5e959b5de9f4f289f92c31cbcb8234a" candidate_uuid: "615b7a2a3e424b8ca8b1182212a0e945" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9ffe0c529954a0f88581546fc6573d3" is_pre_election: true
I20240502 07:58:13.150130 30793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5e959b5de9f4f289f92c31cbcb8234a" candidate_uuid: "615b7a2a3e424b8ca8b1182212a0e945" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" is_pre_election: true
I20240502 07:58:13.150404 30944 raft_consensus.cc:2454] T d5e959b5de9f4f289f92c31cbcb8234a P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 615b7a2a3e424b8ca8b1182212a0e945 in term 0.
I20240502 07:58:13.150404 30793 raft_consensus.cc:2454] T d5e959b5de9f4f289f92c31cbcb8234a P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 615b7a2a3e424b8ca8b1182212a0e945 in term 0.
I20240502 07:58:13.150808 30836 leader_election.cc:302] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 615b7a2a3e424b8ca8b1182212a0e945, 782a2eeace8d4dc68f55d03c1c745d3c, c9ffe0c529954a0f88581546fc6573d3; no voters:
I20240502 07:58:13.150997 30997 raft_consensus.cc:2781] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:13.151057 30997 raft_consensus.cc:482] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:13.151119 30997 raft_consensus.cc:3037] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:13.174409 30997 raft_consensus.cc:504] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:13.174975 30997 leader_election.cc:288] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945 [CANDIDATE]: Term 1 election: Requested vote from peers 782a2eeace8d4dc68f55d03c1c745d3c (127.29.106.129:41727), c9ffe0c529954a0f88581546fc6573d3 (127.29.106.131:34467)
I20240502 07:58:13.175442 30944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5e959b5de9f4f289f92c31cbcb8234a" candidate_uuid: "615b7a2a3e424b8ca8b1182212a0e945" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9ffe0c529954a0f88581546fc6573d3"
I20240502 07:58:13.175442 30793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5e959b5de9f4f289f92c31cbcb8234a" candidate_uuid: "615b7a2a3e424b8ca8b1182212a0e945" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782a2eeace8d4dc68f55d03c1c745d3c"
I20240502 07:58:13.175608 30944 raft_consensus.cc:3037] T d5e959b5de9f4f289f92c31cbcb8234a P c9ffe0c529954a0f88581546fc6573d3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:13.175640 30793 raft_consensus.cc:3037] T d5e959b5de9f4f289f92c31cbcb8234a P 782a2eeace8d4dc68f55d03c1c745d3c [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:13.176635 30793 raft_consensus.cc:2454] T d5e959b5de9f4f289f92c31cbcb8234a P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615b7a2a3e424b8ca8b1182212a0e945 in term 1.
I20240502 07:58:13.176635 30944 raft_consensus.cc:2454] T d5e959b5de9f4f289f92c31cbcb8234a P c9ffe0c529954a0f88581546fc6573d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615b7a2a3e424b8ca8b1182212a0e945 in term 1.
I20240502 07:58:13.177034 30836 leader_election.cc:302] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615b7a2a3e424b8ca8b1182212a0e945, 782a2eeace8d4dc68f55d03c1c745d3c; no voters:
I20240502 07:58:13.177256 30997 raft_consensus.cc:2781] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:13.177336 30997 raft_consensus.cc:686] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 LEADER]: Becoming Leader. State: Replica: 615b7a2a3e424b8ca8b1182212a0e945, State: Running, Role: LEADER
I20240502 07:58:13.177487 30997 consensus_queue.cc:237] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:13.179176 30691 catalog_manager.cc:5484] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945 reported cstate change: term changed from 0 to 1, leader changed from <none> to 615b7a2a3e424b8ca8b1182212a0e945 (127.29.106.130). New cstate: current_term: 1 leader_uuid: "615b7a2a3e424b8ca8b1182212a0e945" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:13.239858 31022 consensus_queue.cc:1035] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:13.242817 31020 consensus_queue.cc:1035] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:13.278303 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:13.280411 31050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:13.281162 31048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:13.281725 31047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:13.282370 30122 server_base.cc:1034] running on GCE node
I20240502 07:58:13.282560 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:58:13.282599 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:58:13.282615 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636693282615 us; error 0 us; skew 500 ppm
I20240502 07:58:13.283814 30122 webserver.cc:460] Webserver started at http://127.29.106.132:37365/ using document root <none> and password file <none>
I20240502 07:58:13.283970 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:13.284034 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:13.284108 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:13.284668 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636688849691-30122-0/minicluster-data/ts-3-root/instance:
uuid: "8c33d607fc1d432eaa1856a40f9ee727"
format_stamp: "Formatted at 2024-05-02 07:58:13 on dist-test-slave-m6l2"
I20240502 07:58:13.286283 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 07:58:13.287334 31055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:13.287631 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:13.287705 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636688849691-30122-0/minicluster-data/ts-3-root
uuid: "8c33d607fc1d432eaa1856a40f9ee727"
format_stamp: "Formatted at 2024-05-02 07:58:13 on dist-test-slave-m6l2"
I20240502 07:58:13.287773 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636688849691-30122-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636688849691-30122-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636688849691-30122-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:13.304540 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:13.304910 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:13.305214 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:13.305734 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:13.305794 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:13.305837 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:13.305857 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:13.311028 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.132:43697
I20240502 07:58:13.311053 31118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.132:43697 every 8 connection(s)
I20240502 07:58:13.315632 31119 heartbeater.cc:347] Connected to a master server at 127.29.106.190:39861
I20240502 07:58:13.315750 31119 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:13.315976 31119 heartbeater.cc:511] Master 127.29.106.190:39861 requested a full tablet report, sending...
I20240502 07:58:13.316663 30690 ts_manager.cc:194] Registered new tserver with Master: 8c33d607fc1d432eaa1856a40f9ee727 (127.29.106.132:43697)
I20240502 07:58:13.317301 30690 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42714
I20240502 07:58:13.354514 30996 consensus_queue.cc:1035] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:13.354518 31020 consensus_queue.cc:1035] T 3175330f9b374e559549707d1107306b P c9ffe0c529954a0f88581546fc6573d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:13.358057 31022 consensus_queue.cc:1035] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:13.358315 31022 consensus_queue.cc:1035] T 3175330f9b374e559549707d1107306b P c9ffe0c529954a0f88581546fc6573d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:13.379828 31125 consensus_queue.cc:1035] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:13.382664 31020 consensus_queue.cc:1035] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:13.443821 31016 consensus_queue.cc:1035] T 46c0c7f5bb734a19a515bf0869de62d6 P 782a2eeace8d4dc68f55d03c1c745d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:13.447048 30997 consensus_queue.cc:1035] T e6509f5318594a9d8e8d8833202652e7 P 615b7a2a3e424b8ca8b1182212a0e945 [LEADER]: Connected to new peer: Peer: permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:13.447145 31016 consensus_queue.cc:1035] T 46c0c7f5bb734a19a515bf0869de62d6 P 782a2eeace8d4dc68f55d03c1c745d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:13.449851 31035 consensus_queue.cc:1035] T e6509f5318594a9d8e8d8833202652e7 P 615b7a2a3e424b8ca8b1182212a0e945 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:13.493364 31035 consensus_queue.cc:1035] T d37ebaee499a4240a6ecfb02aab87b9a P 615b7a2a3e424b8ca8b1182212a0e945 [LEADER]: Connected to new peer: Peer: permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:13.497007 31035 consensus_queue.cc:1035] T d37ebaee499a4240a6ecfb02aab87b9a P 615b7a2a3e424b8ca8b1182212a0e945 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:13.518261 30745 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:58:13.524158 30944 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "229fec0abb91487582fad4da9f264ef5"
dest_uuid: "c9ffe0c529954a0f88581546fc6573d3"
mode: GRACEFUL
new_leader_uuid: "782a2eeace8d4dc68f55d03c1c745d3c"
from {username='slave'} at 127.0.0.1:50298
I20240502 07:58:13.524374 30944 raft_consensus.cc:595] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [term 1 LEADER]: Received request to transfer leadership to TS 782a2eeace8d4dc68f55d03c1c745d3c
I20240502 07:58:13.524829 30745 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240502 07:58:13.525056 30745 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:58:13.574604 31035 consensus_queue.cc:1035] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:13.580685 31035 consensus_queue.cc:1035] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945 [LEADER]: Connected to new peer: Peer: permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:13.935989 31020 raft_consensus.cc:979] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3: : Instructing follower 782a2eeace8d4dc68f55d03c1c745d3c to start an election
I20240502 07:58:13.936153 31125 raft_consensus.cc:1067] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [term 1 LEADER]: Signalling peer 782a2eeace8d4dc68f55d03c1c745d3c to start an election
I20240502 07:58:13.936781 30793 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "229fec0abb91487582fad4da9f264ef5"
dest_uuid: "782a2eeace8d4dc68f55d03c1c745d3c"
from {username='slave'} at 127.0.0.1:39850
I20240502 07:58:13.936950 30793 raft_consensus.cc:482] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:58:13.937018 30793 raft_consensus.cc:3037] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:13.978076 30793 raft_consensus.cc:504] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:13.978588 30793 leader_election.cc:288] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c [CANDIDATE]: Term 2 election: Requested vote from peers 615b7a2a3e424b8ca8b1182212a0e945 (127.29.106.130:34139), c9ffe0c529954a0f88581546fc6573d3 (127.29.106.131:34467)
I20240502 07:58:13.978987 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "229fec0abb91487582fad4da9f264ef5" candidate_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "615b7a2a3e424b8ca8b1182212a0e945"
I20240502 07:58:13.978987 30944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "229fec0abb91487582fad4da9f264ef5" candidate_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c9ffe0c529954a0f88581546fc6573d3"
I20240502 07:58:13.979141 30944 raft_consensus.cc:3032] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:58:13.979209 30944 raft_consensus.cc:729] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c9ffe0c529954a0f88581546fc6573d3, State: Running, Role: LEADER
I20240502 07:58:13.979141 30870 raft_consensus.cc:3037] T 229fec0abb91487582fad4da9f264ef5 P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:13.979401 30944 consensus_queue.cc:260] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:13.979578 30944 raft_consensus.cc:3037] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:13.980309 30870 raft_consensus.cc:2454] T 229fec0abb91487582fad4da9f264ef5 P 615b7a2a3e424b8ca8b1182212a0e945 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 782a2eeace8d4dc68f55d03c1c745d3c in term 2.
I20240502 07:58:13.980382 30944 raft_consensus.cc:2454] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 782a2eeace8d4dc68f55d03c1c745d3c in term 2.
I20240502 07:58:13.980684 30757 leader_election.cc:302] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 782a2eeace8d4dc68f55d03c1c745d3c, c9ffe0c529954a0f88581546fc6573d3; no voters:
I20240502 07:58:13.980935 31016 raft_consensus.cc:2781] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:58:13.981084 31016 raft_consensus.cc:686] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c [term 2 LEADER]: Becoming Leader. State: Replica: 782a2eeace8d4dc68f55d03c1c745d3c, State: Running, Role: LEADER
I20240502 07:58:13.981228 31016 consensus_queue.cc:237] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } }
I20240502 07:58:13.983924 30690 catalog_manager.cc:5484] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c reported cstate change: term changed from 1 to 2, leader changed from c9ffe0c529954a0f88581546fc6573d3 (127.29.106.131) to 782a2eeace8d4dc68f55d03c1c745d3c (127.29.106.129). New cstate: current_term: 2 leader_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:14.318132 31119 heartbeater.cc:503] Master 127.29.106.190:39861 was elected leader, sending a full tablet report...
I20240502 07:58:14.421790 30944 raft_consensus.cc:1261] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [term 2 FOLLOWER]: Refusing update from remote peer 782a2eeace8d4dc68f55d03c1c745d3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 07:58:14.422313 31146 consensus_queue.cc:1035] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 07:58:14.424930 30870 raft_consensus.cc:1261] T 229fec0abb91487582fad4da9f264ef5 P 615b7a2a3e424b8ca8b1182212a0e945 [term 2 FOLLOWER]: Refusing update from remote peer 782a2eeace8d4dc68f55d03c1c745d3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 07:58:14.425333 31146 consensus_queue.cc:1035] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 07:58:15.525533 30745 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:58:15.526772 30745 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 07:58:15.526903 30745 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:58:17.527256 30745 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:58:17.528054 30745 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 07:58:17.528136 30745 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:58:18.534376 30945 consensus_queue.cc:237] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } attrs { replace: true } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } } peers { permanent_uuid: "8c33d607fc1d432eaa1856a40f9ee727" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 43697 } attrs { promote: true } }
I20240502 07:58:18.536474 30792 raft_consensus.cc:1261] T 321c5b918bc14901908bf94c28e0d4e3 P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Refusing update from remote peer c9ffe0c529954a0f88581546fc6573d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:18.536538 30868 raft_consensus.cc:1261] T 321c5b918bc14901908bf94c28e0d4e3 P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 FOLLOWER]: Refusing update from remote peer c9ffe0c529954a0f88581546fc6573d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:18.536900 31156 consensus_queue.cc:1035] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 07:58:18.537868 31154 consensus_queue.cc:1035] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 07:58:18.539861 31160 raft_consensus.cc:2932] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8c33d607fc1d432eaa1856a40f9ee727 (127.29.106.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } attrs { replace: true } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } } peers { permanent_uuid: "8c33d607fc1d432eaa1856a40f9ee727" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 43697 } attrs { promote: true } } }
W20240502 07:58:18.540436 30910 consensus_peers.cc:485] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3 -> Peer 8c33d607fc1d432eaa1856a40f9ee727 (127.29.106.132:43697): Couldn't send request to peer 8c33d607fc1d432eaa1856a40f9ee727. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 321c5b918bc14901908bf94c28e0d4e3. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:18.540619 30792 raft_consensus.cc:2932] T 321c5b918bc14901908bf94c28e0d4e3 P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8c33d607fc1d432eaa1856a40f9ee727 (127.29.106.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } attrs { replace: true } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } } peers { permanent_uuid: "8c33d607fc1d432eaa1856a40f9ee727" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 43697 } attrs { promote: true } } }
I20240502 07:58:18.542884 30690 catalog_manager.cc:5484] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3 reported cstate change: config changed from index -1 to 2, NON_VOTER 8c33d607fc1d432eaa1856a40f9ee727 (127.29.106.132) added. New cstate: current_term: 1 leader_uuid: "c9ffe0c529954a0f88581546fc6573d3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c33d607fc1d432eaa1856a40f9ee727" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 43697 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:18.543385 30868 raft_consensus.cc:2932] T 321c5b918bc14901908bf94c28e0d4e3 P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8c33d607fc1d432eaa1856a40f9ee727 (127.29.106.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } attrs { replace: true } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } } peers { permanent_uuid: "8c33d607fc1d432eaa1856a40f9ee727" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 43697 } attrs { promote: true } } }
I20240502 07:58:18.549388 30868 consensus_queue.cc:237] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } attrs { replace: true } } peers { permanent_uuid: "8c33d607fc1d432eaa1856a40f9ee727" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 43697 } attrs { promote: true } }
I20240502 07:58:18.551023 30792 raft_consensus.cc:1261] T d5e959b5de9f4f289f92c31cbcb8234a P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Refusing update from remote peer 615b7a2a3e424b8ca8b1182212a0e945: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:18.551416 30945 raft_consensus.cc:1261] T d5e959b5de9f4f289f92c31cbcb8234a P c9ffe0c529954a0f88581546fc6573d3 [term 1 FOLLOWER]: Refusing update from remote peer 615b7a2a3e424b8ca8b1182212a0e945: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:18.551626 31172 consensus_queue.cc:1035] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945 [LEADER]: Connected to new peer: Peer: permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 07:58:18.551851 31173 consensus_queue.cc:1035] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 07:58:18.554100 31173 raft_consensus.cc:2932] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8c33d607fc1d432eaa1856a40f9ee727 (127.29.106.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } attrs { replace: true } } peers { permanent_uuid: "8c33d607fc1d432eaa1856a40f9ee727" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 43697 } attrs { promote: true } } }
W20240502 07:58:18.554935 30835 consensus_peers.cc:485] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945 -> Peer 8c33d607fc1d432eaa1856a40f9ee727 (127.29.106.132:43697): Couldn't send request to peer 8c33d607fc1d432eaa1856a40f9ee727. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5e959b5de9f4f289f92c31cbcb8234a. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:18.555094 30792 raft_consensus.cc:2932] T d5e959b5de9f4f289f92c31cbcb8234a P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8c33d607fc1d432eaa1856a40f9ee727 (127.29.106.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } attrs { replace: true } } peers { permanent_uuid: "8c33d607fc1d432eaa1856a40f9ee727" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 43697 } attrs { promote: true } } }
I20240502 07:58:18.556938 30945 raft_consensus.cc:2932] T d5e959b5de9f4f289f92c31cbcb8234a P c9ffe0c529954a0f88581546fc6573d3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8c33d607fc1d432eaa1856a40f9ee727 (127.29.106.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } attrs { replace: true } } peers { permanent_uuid: "8c33d607fc1d432eaa1856a40f9ee727" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 43697 } attrs { promote: true } } }
I20240502 07:58:18.557119 30691 catalog_manager.cc:5484] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945 reported cstate change: config changed from index -1 to 2, NON_VOTER 8c33d607fc1d432eaa1856a40f9ee727 (127.29.106.132) added. New cstate: current_term: 1 leader_uuid: "615b7a2a3e424b8ca8b1182212a0e945" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c33d607fc1d432eaa1856a40f9ee727" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 43697 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:18.558187 30944 consensus_queue.cc:237] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } attrs { replace: true } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } } peers { permanent_uuid: "8c33d607fc1d432eaa1856a40f9ee727" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 43697 } attrs { promote: true } }
I20240502 07:58:18.559654 30868 raft_consensus.cc:1261] T 98055797532e45b3a72d6ef4f278bd76 P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 FOLLOWER]: Refusing update from remote peer c9ffe0c529954a0f88581546fc6573d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:18.559950 30792 raft_consensus.cc:1261] T 98055797532e45b3a72d6ef4f278bd76 P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Refusing update from remote peer c9ffe0c529954a0f88581546fc6573d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 07:58:18.560050 30910 consensus_peers.cc:485] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3 -> Peer 8c33d607fc1d432eaa1856a40f9ee727 (127.29.106.132:43697): Couldn't send request to peer 8c33d607fc1d432eaa1856a40f9ee727. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98055797532e45b3a72d6ef4f278bd76. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:18.560238 31154 consensus_queue.cc:1035] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 07:58:18.560570 31165 consensus_queue.cc:1035] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 07:58:18.562595 31156 raft_consensus.cc:2932] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8c33d607fc1d432eaa1856a40f9ee727 (127.29.106.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } attrs { replace: true } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } } peers { permanent_uuid: "8c33d607fc1d432eaa1856a40f9ee727" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 43697 } attrs { promote: true } } }
I20240502 07:58:18.563091 30792 raft_consensus.cc:2932] T 98055797532e45b3a72d6ef4f278bd76 P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8c33d607fc1d432eaa1856a40f9ee727 (127.29.106.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } attrs { replace: true } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } } peers { permanent_uuid: "8c33d607fc1d432eaa1856a40f9ee727" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 43697 } attrs { promote: true } } }
I20240502 07:58:18.565198 30690 catalog_manager.cc:5484] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3 reported cstate change: config changed from index -1 to 2, NON_VOTER 8c33d607fc1d432eaa1856a40f9ee727 (127.29.106.132) added. New cstate: current_term: 1 leader_uuid: "c9ffe0c529954a0f88581546fc6573d3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c33d607fc1d432eaa1856a40f9ee727" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 43697 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:18.565302 30868 raft_consensus.cc:2932] T 98055797532e45b3a72d6ef4f278bd76 P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8c33d607fc1d432eaa1856a40f9ee727 (127.29.106.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } attrs { replace: true } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } } peers { permanent_uuid: "8c33d607fc1d432eaa1856a40f9ee727" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 43697 } attrs { promote: true } } }
I20240502 07:58:18.571262 30792 consensus_queue.cc:237] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } attrs { replace: true } } peers { permanent_uuid: "8c33d607fc1d432eaa1856a40f9ee727" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 43697 } attrs { promote: true } }
I20240502 07:58:18.573149 30944 raft_consensus.cc:1261] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [term 2 FOLLOWER]: Refusing update from remote peer 782a2eeace8d4dc68f55d03c1c745d3c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 07:58:18.573401 30868 raft_consensus.cc:1261] T 229fec0abb91487582fad4da9f264ef5 P 615b7a2a3e424b8ca8b1182212a0e945 [term 2 FOLLOWER]: Refusing update from remote peer 782a2eeace8d4dc68f55d03c1c745d3c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 07:58:18.573621 31146 consensus_queue.cc:1035] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 07:58:18.573853 31155 consensus_queue.cc:1035] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 07:58:18.575898 31155 raft_consensus.cc:2932] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 8c33d607fc1d432eaa1856a40f9ee727 (127.29.106.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } attrs { replace: true } } peers { permanent_uuid: "8c33d607fc1d432eaa1856a40f9ee727" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 43697 } attrs { promote: true } } }
I20240502 07:58:18.576833 30944 raft_consensus.cc:2932] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 8c33d607fc1d432eaa1856a40f9ee727 (127.29.106.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } attrs { replace: true } } peers { permanent_uuid: "8c33d607fc1d432eaa1856a40f9ee727" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 43697 } attrs { promote: true } } }
I20240502 07:58:18.577993 30868 raft_consensus.cc:2932] T 229fec0abb91487582fad4da9f264ef5 P 615b7a2a3e424b8ca8b1182212a0e945 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 8c33d607fc1d432eaa1856a40f9ee727 (127.29.106.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } attrs { replace: true } } peers { permanent_uuid: "8c33d607fc1d432eaa1856a40f9ee727" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 43697 } attrs { promote: true } } }
I20240502 07:58:18.579010 30689 catalog_manager.cc:5484] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c reported cstate change: config changed from index -1 to 3, NON_VOTER 8c33d607fc1d432eaa1856a40f9ee727 (127.29.106.132) added. New cstate: current_term: 2 leader_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "782a2eeace8d4dc68f55d03c1c745d3c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 41727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "615b7a2a3e424b8ca8b1182212a0e945" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 34139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9ffe0c529954a0f88581546fc6573d3" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c33d607fc1d432eaa1856a40f9ee727" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 43697 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:18.580883 30759 consensus_peers.cc:485] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c -> Peer 8c33d607fc1d432eaa1856a40f9ee727 (127.29.106.132:43697): Couldn't send request to peer 8c33d607fc1d432eaa1856a40f9ee727. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 229fec0abb91487582fad4da9f264ef5. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:18.825073 30122 tablet_server.cc:178] TabletServer@127.29.106.129:0 shutting down...
W20240502 07:58:18.827847 30678 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 2 similar messages]
W20240502 07:58:18.829084 30738 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 07:58:18.829202 30738 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 07:58:18.829900 30122 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 07:58:18.830258 30122 tablet_replica.cc:330] T 3175330f9b374e559549707d1107306b P 782a2eeace8d4dc68f55d03c1c745d3c: stopping tablet replica
I20240502 07:58:18.830400 30122 raft_consensus.cc:2229] T 3175330f9b374e559549707d1107306b P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:18.830550 30122 raft_consensus.cc:2258] T 3175330f9b374e559549707d1107306b P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:18.831056 30122 tablet_replica.cc:330] T d37ebaee499a4240a6ecfb02aab87b9a P 782a2eeace8d4dc68f55d03c1c745d3c: stopping tablet replica
I20240502 07:58:18.831235 30122 raft_consensus.cc:2229] T d37ebaee499a4240a6ecfb02aab87b9a P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:18.831322 30122 raft_consensus.cc:2258] T d37ebaee499a4240a6ecfb02aab87b9a P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:18.831653 30122 tablet_replica.cc:330] T 321c5b918bc14901908bf94c28e0d4e3 P 782a2eeace8d4dc68f55d03c1c745d3c: stopping tablet replica
I20240502 07:58:18.831758 30122 raft_consensus.cc:2229] T 321c5b918bc14901908bf94c28e0d4e3 P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:18.831892 30122 raft_consensus.cc:2258] T 321c5b918bc14901908bf94c28e0d4e3 P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:18.832547 30122 tablet_replica.cc:330] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c: stopping tablet replica
I20240502 07:58:18.832654 30122 raft_consensus.cc:2229] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c [term 2 LEADER]: Raft consensus shutting down.
I20240502 07:58:18.832859 30122 raft_consensus.cc:2258] T 229fec0abb91487582fad4da9f264ef5 P 782a2eeace8d4dc68f55d03c1c745d3c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:18.833356 30122 tablet_replica.cc:330] T e6509f5318594a9d8e8d8833202652e7 P 782a2eeace8d4dc68f55d03c1c745d3c: stopping tablet replica
I20240502 07:58:18.833453 30122 raft_consensus.cc:2229] T e6509f5318594a9d8e8d8833202652e7 P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:18.833549 30122 raft_consensus.cc:2258] T e6509f5318594a9d8e8d8833202652e7 P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:18.833889 30122 tablet_replica.cc:330] T 46c0c7f5bb734a19a515bf0869de62d6 P 782a2eeace8d4dc68f55d03c1c745d3c: stopping tablet replica
I20240502 07:58:18.833992 30122 raft_consensus.cc:2229] T 46c0c7f5bb734a19a515bf0869de62d6 P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 LEADER]: Raft consensus shutting down.
I20240502 07:58:18.834157 30122 raft_consensus.cc:2258] T 46c0c7f5bb734a19a515bf0869de62d6 P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:18.834475 30122 tablet_replica.cc:330] T d5e959b5de9f4f289f92c31cbcb8234a P 782a2eeace8d4dc68f55d03c1c745d3c: stopping tablet replica
I20240502 07:58:18.834573 30122 raft_consensus.cc:2229] T d5e959b5de9f4f289f92c31cbcb8234a P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:18.834683 30122 raft_consensus.cc:2258] T d5e959b5de9f4f289f92c31cbcb8234a P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:18.835125 30122 tablet_replica.cc:330] T 98055797532e45b3a72d6ef4f278bd76 P 782a2eeace8d4dc68f55d03c1c745d3c: stopping tablet replica
I20240502 07:58:18.835228 30122 raft_consensus.cc:2229] T 98055797532e45b3a72d6ef4f278bd76 P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:18.835335 30122 raft_consensus.cc:2258] T 98055797532e45b3a72d6ef4f278bd76 P 782a2eeace8d4dc68f55d03c1c745d3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:18.850833 30122 tablet_server.cc:194] TabletServer@127.29.106.129:0 shutdown complete.
I20240502 07:58:18.854805 30122 tablet_server.cc:178] TabletServer@127.29.106.130:0 shutting down...
W20240502 07:58:18.857364 30738 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 07:58:18.857448 30738 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 07:58:18.861790 30122 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 07:58:18.862126 30122 tablet_replica.cc:330] T 46c0c7f5bb734a19a515bf0869de62d6 P 615b7a2a3e424b8ca8b1182212a0e945: stopping tablet replica
I20240502 07:58:18.862326 30122 raft_consensus.cc:2229] T 46c0c7f5bb734a19a515bf0869de62d6 P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:18.862468 30122 raft_consensus.cc:2258] T 46c0c7f5bb734a19a515bf0869de62d6 P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:18.862983 30122 tablet_replica.cc:330] T 321c5b918bc14901908bf94c28e0d4e3 P 615b7a2a3e424b8ca8b1182212a0e945: stopping tablet replica
I20240502 07:58:18.863072 30122 raft_consensus.cc:2229] T 321c5b918bc14901908bf94c28e0d4e3 P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:18.863252 30122 raft_consensus.cc:2258] T 321c5b918bc14901908bf94c28e0d4e3 P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:18.863775 30122 tablet_replica.cc:330] T 98055797532e45b3a72d6ef4f278bd76 P 615b7a2a3e424b8ca8b1182212a0e945: stopping tablet replica
I20240502 07:58:18.863890 30122 raft_consensus.cc:2229] T 98055797532e45b3a72d6ef4f278bd76 P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:18.864001 30122 raft_consensus.cc:2258] T 98055797532e45b3a72d6ef4f278bd76 P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:18.864570 30122 tablet_replica.cc:330] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945: stopping tablet replica
I20240502 07:58:18.864676 30122 raft_consensus.cc:2229] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 LEADER]: Raft consensus shutting down.
I20240502 07:58:18.864912 30122 raft_consensus.cc:2258] T d5e959b5de9f4f289f92c31cbcb8234a P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:18.865449 30122 tablet_replica.cc:330] T e6509f5318594a9d8e8d8833202652e7 P 615b7a2a3e424b8ca8b1182212a0e945: stopping tablet replica
I20240502 07:58:18.865561 30122 raft_consensus.cc:2229] T e6509f5318594a9d8e8d8833202652e7 P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 LEADER]: Raft consensus shutting down.
I20240502 07:58:18.865717 30122 raft_consensus.cc:2258] T e6509f5318594a9d8e8d8833202652e7 P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:18.866067 30122 tablet_replica.cc:330] T 229fec0abb91487582fad4da9f264ef5 P 615b7a2a3e424b8ca8b1182212a0e945: stopping tablet replica
I20240502 07:58:18.866174 30122 raft_consensus.cc:2229] T 229fec0abb91487582fad4da9f264ef5 P 615b7a2a3e424b8ca8b1182212a0e945 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:18.866286 30122 raft_consensus.cc:2258] T 229fec0abb91487582fad4da9f264ef5 P 615b7a2a3e424b8ca8b1182212a0e945 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:18.866760 30122 tablet_replica.cc:330] T 3175330f9b374e559549707d1107306b P 615b7a2a3e424b8ca8b1182212a0e945: stopping tablet replica
I20240502 07:58:18.866909 30122 raft_consensus.cc:2229] T 3175330f9b374e559549707d1107306b P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:18.867017 30122 raft_consensus.cc:2258] T 3175330f9b374e559549707d1107306b P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:18.867338 30122 tablet_replica.cc:330] T d37ebaee499a4240a6ecfb02aab87b9a P 615b7a2a3e424b8ca8b1182212a0e945: stopping tablet replica
I20240502 07:58:18.867445 30122 raft_consensus.cc:2229] T d37ebaee499a4240a6ecfb02aab87b9a P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 LEADER]: Raft consensus shutting down.
I20240502 07:58:18.867609 30122 raft_consensus.cc:2258] T d37ebaee499a4240a6ecfb02aab87b9a P 615b7a2a3e424b8ca8b1182212a0e945 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 07:58:18.876579 30911 consensus_peers.cc:485] T 3175330f9b374e559549707d1107306b P c9ffe0c529954a0f88581546fc6573d3 -> Peer 782a2eeace8d4dc68f55d03c1c745d3c (127.29.106.129:41727): Couldn't send request to peer 782a2eeace8d4dc68f55d03c1c745d3c. Status: Network error: Client connection negotiation failed: client connection to 127.29.106.129:41727: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:18.882009 30122 tablet_server.cc:194] TabletServer@127.29.106.130:0 shutdown complete.
I20240502 07:58:18.885639 30122 tablet_server.cc:178] TabletServer@127.29.106.131:0 shutting down...
W20240502 07:58:18.887009 30738 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 07:58:18.887104 30738 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 07:58:18.889268 30738 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 07:58:18.889355 30738 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 07:58:18.892191 30122 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 07:58:18.892455 30122 tablet_replica.cc:330] T d5e959b5de9f4f289f92c31cbcb8234a P c9ffe0c529954a0f88581546fc6573d3: stopping tablet replica
I20240502 07:58:18.892601 30122 raft_consensus.cc:2229] T d5e959b5de9f4f289f92c31cbcb8234a P c9ffe0c529954a0f88581546fc6573d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:18.892733 30122 raft_consensus.cc:2258] T d5e959b5de9f4f289f92c31cbcb8234a P c9ffe0c529954a0f88581546fc6573d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:18.893311 30122 tablet_replica.cc:330] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3: stopping tablet replica
I20240502 07:58:18.893399 30122 raft_consensus.cc:2229] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 07:58:18.893574 30122 raft_consensus.cc:2258] T 98055797532e45b3a72d6ef4f278bd76 P c9ffe0c529954a0f88581546fc6573d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:18.894079 30122 tablet_replica.cc:330] T 46c0c7f5bb734a19a515bf0869de62d6 P c9ffe0c529954a0f88581546fc6573d3: stopping tablet replica
I20240502 07:58:18.894173 30122 raft_consensus.cc:2229] T 46c0c7f5bb734a19a515bf0869de62d6 P c9ffe0c529954a0f88581546fc6573d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:18.894250 30122 raft_consensus.cc:2258] T 46c0c7f5bb734a19a515bf0869de62d6 P c9ffe0c529954a0f88581546fc6573d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:18.894562 30122 tablet_replica.cc:330] T d37ebaee499a4240a6ecfb02aab87b9a P c9ffe0c529954a0f88581546fc6573d3: stopping tablet replica
I20240502 07:58:18.894639 30122 raft_consensus.cc:2229] T d37ebaee499a4240a6ecfb02aab87b9a P c9ffe0c529954a0f88581546fc6573d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:18.894708 30122 raft_consensus.cc:2258] T d37ebaee499a4240a6ecfb02aab87b9a P c9ffe0c529954a0f88581546fc6573d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:18.894953 30122 tablet_replica.cc:330] T e6509f5318594a9d8e8d8833202652e7 P c9ffe0c529954a0f88581546fc6573d3: stopping tablet replica
I20240502 07:58:18.895028 30122 raft_consensus.cc:2229] T e6509f5318594a9d8e8d8833202652e7 P c9ffe0c529954a0f88581546fc6573d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:18.895098 30122 raft_consensus.cc:2258] T e6509f5318594a9d8e8d8833202652e7 P c9ffe0c529954a0f88581546fc6573d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:18.895351 30122 tablet_replica.cc:330] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3: stopping tablet replica
I20240502 07:58:18.895426 30122 raft_consensus.cc:2229] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:18.895502 30122 raft_consensus.cc:2258] T 229fec0abb91487582fad4da9f264ef5 P c9ffe0c529954a0f88581546fc6573d3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:18.895941 30122 tablet_replica.cc:330] T 3175330f9b374e559549707d1107306b P c9ffe0c529954a0f88581546fc6573d3: stopping tablet replica
I20240502 07:58:18.896014 30122 raft_consensus.cc:2229] T 3175330f9b374e559549707d1107306b P c9ffe0c529954a0f88581546fc6573d3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 07:58:18.896169 30122 raft_consensus.cc:2258] T 3175330f9b374e559549707d1107306b P c9ffe0c529954a0f88581546fc6573d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:18.896474 30122 tablet_replica.cc:330] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3: stopping tablet replica
I20240502 07:58:18.896544 30122 raft_consensus.cc:2229] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 07:58:18.896687 30122 raft_consensus.cc:2258] T 321c5b918bc14901908bf94c28e0d4e3 P c9ffe0c529954a0f88581546fc6573d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:18.911463 30122 tablet_server.cc:194] TabletServer@127.29.106.131:0 shutdown complete.
I20240502 07:58:18.914927 30122 tablet_server.cc:178] TabletServer@127.29.106.132:0 shutting down...
I20240502 07:58:18.919450 30122 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 07:58:18.932663 30122 tablet_server.cc:194] TabletServer@127.29.106.132:0 shutdown complete.
I20240502 07:58:18.935052 30122 master.cc:537] Master@127.29.106.190:39861 shutting down...
I20240502 07:58:18.939030 30122 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fe84c8508f9643309015ae12d6058dbb [term 1 LEADER]: Raft consensus shutting down.
I20240502 07:58:18.939244 30122 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fe84c8508f9643309015ae12d6058dbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:18.939306 30122 tablet_replica.cc:330] T 00000000000000000000000000000000 P fe84c8508f9643309015ae12d6058dbb: stopping tablet replica
I20240502 07:58:18.942574 30122 master.cc:552] Master@127.29.106.190:39861 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (7476 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20240502 07:58:18.955312 30122 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.106.190:43859,127.29.106.189:40887,127.29.106.188:38279
I20240502 07:58:18.955766 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:18.959015 31189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:18.959941 30122 server_base.cc:1034] running on GCE node
W20240502 07:58:18.960772 31188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:18.961266 31191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:18.961520 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:58:18.961582 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:58:18.961608 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636698961607 us; error 0 us; skew 500 ppm
I20240502 07:58:18.962779 30122 webserver.cc:460] Webserver started at http://127.29.106.190:41313/ using document root <none> and password file <none>
I20240502 07:58:18.962961 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:18.963029 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:18.963119 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:18.963591 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636688849691-30122-0/minicluster-data/master-0-root/instance:
uuid: "e3d8e51c20fd46d18c090295db4fca94"
format_stamp: "Formatted at 2024-05-02 07:58:18 on dist-test-slave-m6l2"
I20240502 07:58:18.965369 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:18.966635 31196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:18.967037 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:18.967144 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636688849691-30122-0/minicluster-data/master-0-root
uuid: "e3d8e51c20fd46d18c090295db4fca94"
format_stamp: "Formatted at 2024-05-02 07:58:18 on dist-test-slave-m6l2"
I20240502 07:58:18.967255 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636688849691-30122-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636688849691-30122-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636688849691-30122-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:18.980495 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:18.980969 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:18.985752 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.190:43859
I20240502 07:58:18.987454 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 07:58:18.992898 31248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.190:43859 every 8 connection(s)
I20240502 07:58:18.993605 31249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 07:58:18.994781 31252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:18.995128 31249 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 } has no permanent_uuid. Determining permanent_uuid...
W20240502 07:58:18.995513 31251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:18.996433 31254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:18.997474 30122 server_base.cc:1034] running on GCE node
I20240502 07:58:18.997658 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:58:18.997699 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:58:18.997730 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636698997728 us; error 0 us; skew 500 ppm
I20240502 07:58:18.998962 30122 webserver.cc:460] Webserver started at http://127.29.106.189:34815/ using document root <none> and password file <none>
I20240502 07:58:18.999171 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:18.999230 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:18.999310 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:18.999785 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636688849691-30122-0/minicluster-data/master-1-root/instance:
uuid: "1298d969e51c4856884d33ce860464b6"
format_stamp: "Formatted at 2024-05-02 07:58:18 on dist-test-slave-m6l2"
I20240502 07:58:19.000900 31249 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.189" port: 40887 } has no permanent_uuid. Determining permanent_uuid...
I20240502 07:58:19.001616 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
W20240502 07:58:19.002457 31249 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.29.106.189:40887: Network error: Client connection negotiation failed: client connection to 127.29.106.189:40887: connect: Connection refused (error 111)
I20240502 07:58:19.002895 31262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:19.003185 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:19.003259 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636688849691-30122-0/minicluster-data/master-1-root
uuid: "1298d969e51c4856884d33ce860464b6"
format_stamp: "Formatted at 2024-05-02 07:58:18 on dist-test-slave-m6l2"
I20240502 07:58:19.003342 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636688849691-30122-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636688849691-30122-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636688849691-30122-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:19.015290 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:19.015764 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:19.020696 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.189:40887
I20240502 07:58:19.020700 31314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.189:40887 every 8 connection(s)
I20240502 07:58:19.021831 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 07:58:19.022094 31315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:19.023779 31315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 } has no permanent_uuid. Determining permanent_uuid...
W20240502 07:58:19.024611 31317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:19.025262 31318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:19.026275 31320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:19.026886 30122 server_base.cc:1034] running on GCE node
I20240502 07:58:19.027318 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:58:19.027388 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:58:19.027413 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636699027412 us; error 0 us; skew 500 ppm
I20240502 07:58:19.028419 30122 webserver.cc:460] Webserver started at http://127.29.106.188:40677/ using document root <none> and password file <none>
I20240502 07:58:19.028581 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:19.028656 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:19.028743 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:19.028807 31315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.189" port: 40887 } has no permanent_uuid. Determining permanent_uuid...
I20240502 07:58:19.029173 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636688849691-30122-0/minicluster-data/master-2-root/instance:
uuid: "c8d8e32856424b588814b81dff25aa25"
format_stamp: "Formatted at 2024-05-02 07:58:19 on dist-test-slave-m6l2"
I20240502 07:58:19.030972 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:19.032172 31327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:19.032559 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:19.032650 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636688849691-30122-0/minicluster-data/master-2-root
uuid: "c8d8e32856424b588814b81dff25aa25"
format_stamp: "Formatted at 2024-05-02 07:58:19 on dist-test-slave-m6l2"
I20240502 07:58:19.032754 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636688849691-30122-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636688849691-30122-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636688849691-30122-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:19.033843 31315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } has no permanent_uuid. Determining permanent_uuid...
W20240502 07:58:19.035068 31315 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.29.106.188:38279: Network error: Client connection negotiation failed: client connection to 127.29.106.188:38279: connect: Connection refused (error 111)
I20240502 07:58:19.059424 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:19.059983 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:19.061676 31249 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.189" port: 40887 } attempt: 1
I20240502 07:58:19.064253 31315 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } attempt: 1
I20240502 07:58:19.065526 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.188:38279
W20240502 07:58:19.065569 31315 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.29.106.188:38279: Network error: Client connection negotiation failed: client connection to 127.29.106.188:38279: connect: Connection refused (error 111)
I20240502 07:58:19.065752 31380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.188:38279 every 8 connection(s)
I20240502 07:58:19.066110 30122 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 07:58:19.066637 31381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:19.067133 31249 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } has no permanent_uuid. Determining permanent_uuid...
I20240502 07:58:19.068629 31381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 } has no permanent_uuid. Determining permanent_uuid...
I20240502 07:58:19.073577 31249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94: Bootstrap starting.
I20240502 07:58:19.074491 31249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:19.075681 31381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.189" port: 40887 } has no permanent_uuid. Determining permanent_uuid...
I20240502 07:58:19.075757 31249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94: No bootstrap required, opened a new log
I20240502 07:58:19.076200 31249 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8e51c20fd46d18c090295db4fca94" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 } } peers { permanent_uuid: "1298d969e51c4856884d33ce860464b6" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 40887 } } peers { permanent_uuid: "c8d8e32856424b588814b81dff25aa25" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } }
I20240502 07:58:19.076370 31249 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:19.076417 31249 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3d8e51c20fd46d18c090295db4fca94, State: Initialized, Role: FOLLOWER
I20240502 07:58:19.076562 31249 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8e51c20fd46d18c090295db4fca94" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 } } peers { permanent_uuid: "1298d969e51c4856884d33ce860464b6" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 40887 } } peers { permanent_uuid: "c8d8e32856424b588814b81dff25aa25" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } }
I20240502 07:58:19.077078 31249 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 07:58:19.077394 31386 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8e51c20fd46d18c090295db4fca94" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 } } peers { permanent_uuid: "1298d969e51c4856884d33ce860464b6" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 40887 } } peers { permanent_uuid: "c8d8e32856424b588814b81dff25aa25" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } } }
I20240502 07:58:19.077497 31386 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 07:58:19.079387 30122 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20240502 07:58:19.079762 31397 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 07:58:19.079844 31397 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 07:58:19.080055 31381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } has no permanent_uuid. Determining permanent_uuid...
I20240502 07:58:19.085289 31381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25: Bootstrap starting.
I20240502 07:58:19.086228 31381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:19.087489 31381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25: No bootstrap required, opened a new log
I20240502 07:58:19.087919 31381 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8e51c20fd46d18c090295db4fca94" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 } } peers { permanent_uuid: "1298d969e51c4856884d33ce860464b6" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 40887 } } peers { permanent_uuid: "c8d8e32856424b588814b81dff25aa25" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } }
I20240502 07:58:19.088032 31381 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:19.088055 31381 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d8e32856424b588814b81dff25aa25, State: Initialized, Role: FOLLOWER
I20240502 07:58:19.088179 31381 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8e51c20fd46d18c090295db4fca94" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 } } peers { permanent_uuid: "1298d969e51c4856884d33ce860464b6" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 40887 } } peers { permanent_uuid: "c8d8e32856424b588814b81dff25aa25" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } }
I20240502 07:58:19.088609 31399 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8e51c20fd46d18c090295db4fca94" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 } } peers { permanent_uuid: "1298d969e51c4856884d33ce860464b6" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 40887 } } peers { permanent_uuid: "c8d8e32856424b588814b81dff25aa25" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } } }
I20240502 07:58:19.088697 31381 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 07:58:19.088784 31399 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [sys.catalog]: This master's current role is: FOLLOWER
W20240502 07:58:19.091147 31410 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 07:58:19.091228 31410 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 07:58:19.138814 31315 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } attempt: 2
I20240502 07:58:19.144560 31315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6: Bootstrap starting.
I20240502 07:58:19.145494 31315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:19.146793 31315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6: No bootstrap required, opened a new log
I20240502 07:58:19.147195 31315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8e51c20fd46d18c090295db4fca94" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 } } peers { permanent_uuid: "1298d969e51c4856884d33ce860464b6" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 40887 } } peers { permanent_uuid: "c8d8e32856424b588814b81dff25aa25" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } }
I20240502 07:58:19.147303 31315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:19.147326 31315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1298d969e51c4856884d33ce860464b6, State: Initialized, Role: FOLLOWER
I20240502 07:58:19.147446 31315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8e51c20fd46d18c090295db4fca94" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 } } peers { permanent_uuid: "1298d969e51c4856884d33ce860464b6" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 40887 } } peers { permanent_uuid: "c8d8e32856424b588814b81dff25aa25" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } }
I20240502 07:58:19.147873 31315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 07:58:19.147804 31412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8e51c20fd46d18c090295db4fca94" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 } } peers { permanent_uuid: "1298d969e51c4856884d33ce860464b6" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 40887 } } peers { permanent_uuid: "c8d8e32856424b588814b81dff25aa25" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } } }
I20240502 07:58:19.147976 31412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 07:58:19.149643 30122 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240502 07:58:19.150437 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:19.150555 31423 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 07:58:19.150619 31423 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 07:58:19.153389 31424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:19.154021 31427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:19.154997 31425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:19.155658 30122 server_base.cc:1034] running on GCE node
I20240502 07:58:19.155840 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:58:19.155881 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:58:19.155898 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636699155898 us; error 0 us; skew 500 ppm
I20240502 07:58:19.157235 30122 webserver.cc:460] Webserver started at http://127.29.106.129:34869/ using document root <none> and password file <none>
I20240502 07:58:19.157385 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:19.157438 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:19.157495 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:19.157861 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636688849691-30122-0/minicluster-data/ts-0-root/instance:
uuid: "77ec61f2c97b4db9983578fe9d587dba"
format_stamp: "Formatted at 2024-05-02 07:58:19 on dist-test-slave-m6l2"
I20240502 07:58:19.159648 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:19.160701 31432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:19.161093 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:19.161170 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636688849691-30122-0/minicluster-data/ts-0-root
uuid: "77ec61f2c97b4db9983578fe9d587dba"
format_stamp: "Formatted at 2024-05-02 07:58:19 on dist-test-slave-m6l2"
I20240502 07:58:19.161273 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636688849691-30122-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636688849691-30122-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636688849691-30122-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:19.171311 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:19.171929 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:19.172392 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:19.172963 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:19.173012 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:19.173048 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:19.173063 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:19.179051 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.129:42199
I20240502 07:58:19.179152 31495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.129:42199 every 8 connection(s)
I20240502 07:58:19.180341 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:19.184605 31506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:19.185410 31497 heartbeater.cc:347] Connected to a master server at 127.29.106.190:43859
I20240502 07:58:19.185532 31497 heartbeater.cc:464] Registering TS with master...
W20240502 07:58:19.185686 31503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:19.185875 31497 heartbeater.cc:511] Master 127.29.106.190:43859 requested a full tablet report, sending...
W20240502 07:58:19.186179 31504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:19.186367 31496 heartbeater.cc:347] Connected to a master server at 127.29.106.188:38279
I20240502 07:58:19.186446 31496 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:19.186692 30122 server_base.cc:1034] running on GCE node
I20240502 07:58:19.186770 31496 heartbeater.cc:511] Master 127.29.106.188:38279 requested a full tablet report, sending...
I20240502 07:58:19.186937 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240502 07:58:19.186954 31213 ts_manager.cc:194] Registered new tserver with Master: 77ec61f2c97b4db9983578fe9d587dba (127.29.106.129:42199)
W20240502 07:58:19.186982 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:58:19.187103 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636699187100 us; error 0 us; skew 500 ppm
I20240502 07:58:19.187548 31498 heartbeater.cc:347] Connected to a master server at 127.29.106.189:40887
I20240502 07:58:19.187618 31345 ts_manager.cc:194] Registered new tserver with Master: 77ec61f2c97b4db9983578fe9d587dba (127.29.106.129:42199)
I20240502 07:58:19.187629 31498 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:19.187903 31498 heartbeater.cc:511] Master 127.29.106.189:40887 requested a full tablet report, sending...
I20240502 07:58:19.188562 30122 webserver.cc:460] Webserver started at http://127.29.106.130:39745/ using document root <none> and password file <none>
I20240502 07:58:19.188606 31279 ts_manager.cc:194] Registered new tserver with Master: 77ec61f2c97b4db9983578fe9d587dba (127.29.106.129:42199)
I20240502 07:58:19.188725 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:19.188786 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:19.188874 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:19.189294 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636688849691-30122-0/minicluster-data/ts-1-root/instance:
uuid: "6716d6d1cba44ec1bcc584707531fba1"
format_stamp: "Formatted at 2024-05-02 07:58:19 on dist-test-slave-m6l2"
I20240502 07:58:19.190853 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 07:58:19.191917 31511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:19.192513 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:19.192602 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636688849691-30122-0/minicluster-data/ts-1-root
uuid: "6716d6d1cba44ec1bcc584707531fba1"
format_stamp: "Formatted at 2024-05-02 07:58:19 on dist-test-slave-m6l2"
I20240502 07:58:19.192682 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636688849691-30122-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636688849691-30122-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636688849691-30122-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:19.201867 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:19.202353 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:19.202662 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:19.203194 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:19.203236 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:19.203284 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:19.203320 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:19.210007 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.130:44715
I20240502 07:58:19.210112 31574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.130:44715 every 8 connection(s)
I20240502 07:58:19.211141 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:19.214871 31583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:19.215484 31585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:19.216609 31577 heartbeater.cc:347] Connected to a master server at 127.29.106.189:40887
W20240502 07:58:19.216637 31582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:19.216714 31577 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:19.217090 31576 heartbeater.cc:347] Connected to a master server at 127.29.106.190:43859
I20240502 07:58:19.217164 31577 heartbeater.cc:511] Master 127.29.106.189:40887 requested a full tablet report, sending...
I20240502 07:58:19.217206 31576 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:19.217309 30122 server_base.cc:1034] running on GCE node
I20240502 07:58:19.217497 31576 heartbeater.cc:511] Master 127.29.106.190:43859 requested a full tablet report, sending...
I20240502 07:58:19.217521 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:58:19.217558 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:58:19.217581 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636699217580 us; error 0 us; skew 500 ppm
I20240502 07:58:19.217588 31575 heartbeater.cc:347] Connected to a master server at 127.29.106.188:38279
I20240502 07:58:19.217672 31575 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:19.217968 31575 heartbeater.cc:511] Master 127.29.106.188:38279 requested a full tablet report, sending...
I20240502 07:58:19.218005 31279 ts_manager.cc:194] Registered new tserver with Master: 6716d6d1cba44ec1bcc584707531fba1 (127.29.106.130:44715)
I20240502 07:58:19.218701 31345 ts_manager.cc:194] Registered new tserver with Master: 6716d6d1cba44ec1bcc584707531fba1 (127.29.106.130:44715)
I20240502 07:58:19.218789 31213 ts_manager.cc:194] Registered new tserver with Master: 6716d6d1cba44ec1bcc584707531fba1 (127.29.106.130:44715)
I20240502 07:58:19.218830 30122 webserver.cc:460] Webserver started at http://127.29.106.131:44633/ using document root <none> and password file <none>
I20240502 07:58:19.218998 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:19.219094 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:19.219187 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:19.219650 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636688849691-30122-0/minicluster-data/ts-2-root/instance:
uuid: "be2413644ddc406da9fd02b7bd712f09"
format_stamp: "Formatted at 2024-05-02 07:58:19 on dist-test-slave-m6l2"
I20240502 07:58:19.221649 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240502 07:58:19.222941 31590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:19.223292 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:19.223377 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636688849691-30122-0/minicluster-data/ts-2-root
uuid: "be2413644ddc406da9fd02b7bd712f09"
format_stamp: "Formatted at 2024-05-02 07:58:19 on dist-test-slave-m6l2"
I20240502 07:58:19.223459 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636688849691-30122-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636688849691-30122-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636688849691-30122-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:19.245692 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:19.246261 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:19.246656 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:19.247198 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:19.247241 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:19.247301 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:19.247334 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:19.253367 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.131:43521
I20240502 07:58:19.253450 31653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.131:43521 every 8 connection(s)
I20240502 07:58:19.259696 31654 heartbeater.cc:347] Connected to a master server at 127.29.106.188:38279
I20240502 07:58:19.259832 31654 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:19.260125 31654 heartbeater.cc:511] Master 127.29.106.188:38279 requested a full tablet report, sending...
I20240502 07:58:19.260855 31345 ts_manager.cc:194] Registered new tserver with Master: be2413644ddc406da9fd02b7bd712f09 (127.29.106.131:43521)
I20240502 07:58:19.262279 31656 heartbeater.cc:347] Connected to a master server at 127.29.106.189:40887
I20240502 07:58:19.262372 31656 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:19.262614 31656 heartbeater.cc:511] Master 127.29.106.189:40887 requested a full tablet report, sending...
I20240502 07:58:19.263154 31279 ts_manager.cc:194] Registered new tserver with Master: be2413644ddc406da9fd02b7bd712f09 (127.29.106.131:43521)
I20240502 07:58:19.272126 31655 heartbeater.cc:347] Connected to a master server at 127.29.106.190:43859
I20240502 07:58:19.272318 31655 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:19.272590 31655 heartbeater.cc:511] Master 127.29.106.190:43859 requested a full tablet report, sending...
I20240502 07:58:19.273445 31213 ts_manager.cc:194] Registered new tserver with Master: be2413644ddc406da9fd02b7bd712f09 (127.29.106.131:43521)
I20240502 07:58:19.273881 30122 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.019756666s
I20240502 07:58:19.359249 31412 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:19.359432 31412 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8e51c20fd46d18c090295db4fca94" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 } } peers { permanent_uuid: "1298d969e51c4856884d33ce860464b6" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 40887 } } peers { permanent_uuid: "c8d8e32856424b588814b81dff25aa25" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } }
I20240502 07:58:19.359915 31412 leader_election.cc:288] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3d8e51c20fd46d18c090295db4fca94 (127.29.106.190:43859), c8d8e32856424b588814b81dff25aa25 (127.29.106.188:38279)
I20240502 07:58:19.360435 31223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1298d969e51c4856884d33ce860464b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3d8e51c20fd46d18c090295db4fca94" is_pre_election: true
I20240502 07:58:19.360440 31355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1298d969e51c4856884d33ce860464b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d8e32856424b588814b81dff25aa25" is_pre_election: true
I20240502 07:58:19.360567 31223 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1298d969e51c4856884d33ce860464b6 in term 0.
I20240502 07:58:19.360580 31355 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1298d969e51c4856884d33ce860464b6 in term 0.
I20240502 07:58:19.360875 31264 leader_election.cc:302] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1298d969e51c4856884d33ce860464b6, e3d8e51c20fd46d18c090295db4fca94; no voters:
I20240502 07:58:19.361025 31412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:19.361084 31412 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:19.361141 31412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:19.362107 31412 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8e51c20fd46d18c090295db4fca94" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 } } peers { permanent_uuid: "1298d969e51c4856884d33ce860464b6" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 40887 } } peers { permanent_uuid: "c8d8e32856424b588814b81dff25aa25" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } }
I20240502 07:58:19.362447 31412 leader_election.cc:288] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6 [CANDIDATE]: Term 1 election: Requested vote from peers e3d8e51c20fd46d18c090295db4fca94 (127.29.106.190:43859), c8d8e32856424b588814b81dff25aa25 (127.29.106.188:38279)
I20240502 07:58:19.362833 31223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1298d969e51c4856884d33ce860464b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3d8e51c20fd46d18c090295db4fca94"
I20240502 07:58:19.362861 31355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1298d969e51c4856884d33ce860464b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d8e32856424b588814b81dff25aa25"
I20240502 07:58:19.362951 31223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:19.362978 31355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:19.363782 31223 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1298d969e51c4856884d33ce860464b6 in term 1.
I20240502 07:58:19.363838 31355 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1298d969e51c4856884d33ce860464b6 in term 1.
I20240502 07:58:19.364080 31263 leader_election.cc:302] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1298d969e51c4856884d33ce860464b6, c8d8e32856424b588814b81dff25aa25; no voters:
I20240502 07:58:19.364252 31412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:19.364497 31412 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6 [term 1 LEADER]: Becoming Leader. State: Replica: 1298d969e51c4856884d33ce860464b6, State: Running, Role: LEADER
I20240502 07:58:19.364650 31412 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8e51c20fd46d18c090295db4fca94" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 } } peers { permanent_uuid: "1298d969e51c4856884d33ce860464b6" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 40887 } } peers { permanent_uuid: "c8d8e32856424b588814b81dff25aa25" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } }
I20240502 07:58:19.365341 31661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1298d969e51c4856884d33ce860464b6. Latest consensus state: current_term: 1 leader_uuid: "1298d969e51c4856884d33ce860464b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8e51c20fd46d18c090295db4fca94" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 } } peers { permanent_uuid: "1298d969e51c4856884d33ce860464b6" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 40887 } } peers { permanent_uuid: "c8d8e32856424b588814b81dff25aa25" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } } }
I20240502 07:58:19.365465 31661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6 [sys.catalog]: This master's current role is: LEADER
I20240502 07:58:19.365794 31663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 07:58:19.366554 31663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 07:58:19.368393 31223 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94 [term 1 FOLLOWER]: Refusing update from remote peer 1298d969e51c4856884d33ce860464b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:19.368400 31355 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [term 1 FOLLOWER]: Refusing update from remote peer 1298d969e51c4856884d33ce860464b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:19.368752 31412 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3d8e51c20fd46d18c090295db4fca94" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:19.368873 31661 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d8e32856424b588814b81dff25aa25" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:19.370131 31386 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1298d969e51c4856884d33ce860464b6. Latest consensus state: current_term: 1 leader_uuid: "1298d969e51c4856884d33ce860464b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8e51c20fd46d18c090295db4fca94" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 } } peers { permanent_uuid: "1298d969e51c4856884d33ce860464b6" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 40887 } } peers { permanent_uuid: "c8d8e32856424b588814b81dff25aa25" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } } }
I20240502 07:58:19.370236 31386 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 07:58:19.370183 31399 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1298d969e51c4856884d33ce860464b6. Latest consensus state: current_term: 1 leader_uuid: "1298d969e51c4856884d33ce860464b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8e51c20fd46d18c090295db4fca94" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 } } peers { permanent_uuid: "1298d969e51c4856884d33ce860464b6" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 40887 } } peers { permanent_uuid: "c8d8e32856424b588814b81dff25aa25" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } } }
I20240502 07:58:19.370285 31399 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 07:58:19.371418 31412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1298d969e51c4856884d33ce860464b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8e51c20fd46d18c090295db4fca94" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 } } peers { permanent_uuid: "1298d969e51c4856884d33ce860464b6" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 40887 } } peers { permanent_uuid: "c8d8e32856424b588814b81dff25aa25" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } } }
I20240502 07:58:19.371546 31412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6 [sys.catalog]: This master's current role is: LEADER
I20240502 07:58:19.371659 31412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1298d969e51c4856884d33ce860464b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8e51c20fd46d18c090295db4fca94" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 } } peers { permanent_uuid: "1298d969e51c4856884d33ce860464b6" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 40887 } } peers { permanent_uuid: "c8d8e32856424b588814b81dff25aa25" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } } }
I20240502 07:58:19.371731 31412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6 [sys.catalog]: This master's current role is: LEADER
I20240502 07:58:19.372056 31399 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1298d969e51c4856884d33ce860464b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8e51c20fd46d18c090295db4fca94" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 } } peers { permanent_uuid: "1298d969e51c4856884d33ce860464b6" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 40887 } } peers { permanent_uuid: "c8d8e32856424b588814b81dff25aa25" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } } }
I20240502 07:58:19.372121 31399 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 07:58:19.372756 31386 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1298d969e51c4856884d33ce860464b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8e51c20fd46d18c090295db4fca94" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 } } peers { permanent_uuid: "1298d969e51c4856884d33ce860464b6" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 40887 } } peers { permanent_uuid: "c8d8e32856424b588814b81dff25aa25" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } } }
I20240502 07:58:19.372893 31386 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 07:58:19.372915 31663 catalog_manager.cc:1347] Generated new cluster ID: c3223e60d99445a78d005007160aceb2
I20240502 07:58:19.372952 31663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 07:58:19.390564 31663 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 07:58:19.391564 31663 catalog_manager.cc:1504] Loading token signing keys...
I20240502 07:58:19.399360 31663 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6: Generated new TSK 0
I20240502 07:58:19.399578 31663 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 07:58:20.080763 31397 catalog_manager.cc:1259] Loaded cluster ID: c3223e60d99445a78d005007160aceb2
I20240502 07:58:20.080852 31397 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94: loading cluster ID for follower catalog manager: success
I20240502 07:58:20.082051 31397 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94: acquiring CA information for follower catalog manager: success
I20240502 07:58:20.082623 31397 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 07:58:20.091976 31410 catalog_manager.cc:1259] Loaded cluster ID: c3223e60d99445a78d005007160aceb2
I20240502 07:58:20.092057 31410 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25: loading cluster ID for follower catalog manager: success
I20240502 07:58:20.093375 31410 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25: acquiring CA information for follower catalog manager: success
I20240502 07:58:20.093878 31410 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 07:58:20.190585 31279 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56028
I20240502 07:58:20.220528 31279 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56044
I20240502 07:58:20.264757 31279 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56048
I20240502 07:58:21.149816 31422 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:58:21.191525 31498 heartbeater.cc:503] Master 127.29.106.189:40887 was elected leader, sending a full tablet report...
I20240502 07:58:21.221449 31577 heartbeater.cc:503] Master 127.29.106.189:40887 was elected leader, sending a full tablet report...
I20240502 07:58:21.265725 31656 heartbeater.cc:503] Master 127.29.106.189:40887 was elected leader, sending a full tablet report...
I20240502 07:58:21.529484 30122 test_util.cc:268] Using random seed: -142379639
I20240502 07:58:21.540705 31279 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56062:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 07:58:21.557667 31539 tablet_service.cc:1467] Processing CreateTablet for tablet 5ceb582179914d678d162c4320c3eae3 (DEFAULT_TABLE table=test-workload [id=eb42958ed31848a0845250bbee3ce6e8]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240502 07:58:21.557686 31460 tablet_service.cc:1467] Processing CreateTablet for tablet 9a17a897c6f744f09e7c50101b773a40 (DEFAULT_TABLE table=test-workload [id=eb42958ed31848a0845250bbee3ce6e8]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240502 07:58:21.558079 31539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ceb582179914d678d162c4320c3eae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:21.558125 31460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a17a897c6f744f09e7c50101b773a40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:21.561280 31701 tablet_bootstrap.cc:492] T 9a17a897c6f744f09e7c50101b773a40 P 77ec61f2c97b4db9983578fe9d587dba: Bootstrap starting.
I20240502 07:58:21.561297 31702 tablet_bootstrap.cc:492] T 5ceb582179914d678d162c4320c3eae3 P 6716d6d1cba44ec1bcc584707531fba1: Bootstrap starting.
I20240502 07:58:21.561815 31618 tablet_service.cc:1467] Processing CreateTablet for tablet f0b23d51073649cea865478bb091bdf2 (DEFAULT_TABLE table=test-workload [id=eb42958ed31848a0845250bbee3ce6e8]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240502 07:58:21.562108 31618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b23d51073649cea865478bb091bdf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:21.562557 31702 tablet_bootstrap.cc:654] T 5ceb582179914d678d162c4320c3eae3 P 6716d6d1cba44ec1bcc584707531fba1: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:21.562675 31701 tablet_bootstrap.cc:654] T 9a17a897c6f744f09e7c50101b773a40 P 77ec61f2c97b4db9983578fe9d587dba: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:21.564118 31701 tablet_bootstrap.cc:492] T 9a17a897c6f744f09e7c50101b773a40 P 77ec61f2c97b4db9983578fe9d587dba: No bootstrap required, opened a new log
I20240502 07:58:21.564220 31701 ts_tablet_manager.cc:1367] T 9a17a897c6f744f09e7c50101b773a40 P 77ec61f2c97b4db9983578fe9d587dba: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:21.564543 31705 tablet_bootstrap.cc:492] T f0b23d51073649cea865478bb091bdf2 P be2413644ddc406da9fd02b7bd712f09: Bootstrap starting.
I20240502 07:58:21.564888 31701 raft_consensus.cc:348] T 9a17a897c6f744f09e7c50101b773a40 P 77ec61f2c97b4db9983578fe9d587dba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77ec61f2c97b4db9983578fe9d587dba" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42199 } }
I20240502 07:58:21.564998 31701 raft_consensus.cc:374] T 9a17a897c6f744f09e7c50101b773a40 P 77ec61f2c97b4db9983578fe9d587dba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:21.565037 31701 raft_consensus.cc:729] T 9a17a897c6f744f09e7c50101b773a40 P 77ec61f2c97b4db9983578fe9d587dba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ec61f2c97b4db9983578fe9d587dba, State: Initialized, Role: FOLLOWER
I20240502 07:58:21.565219 31701 consensus_queue.cc:260] T 9a17a897c6f744f09e7c50101b773a40 P 77ec61f2c97b4db9983578fe9d587dba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77ec61f2c97b4db9983578fe9d587dba" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42199 } }
I20240502 07:58:21.565299 31701 raft_consensus.cc:388] T 9a17a897c6f744f09e7c50101b773a40 P 77ec61f2c97b4db9983578fe9d587dba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:58:21.565357 31701 raft_consensus.cc:482] T 9a17a897c6f744f09e7c50101b773a40 P 77ec61f2c97b4db9983578fe9d587dba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:58:21.565413 31701 raft_consensus.cc:3037] T 9a17a897c6f744f09e7c50101b773a40 P 77ec61f2c97b4db9983578fe9d587dba [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:21.565554 31702 tablet_bootstrap.cc:492] T 5ceb582179914d678d162c4320c3eae3 P 6716d6d1cba44ec1bcc584707531fba1: No bootstrap required, opened a new log
I20240502 07:58:21.565595 31705 tablet_bootstrap.cc:654] T f0b23d51073649cea865478bb091bdf2 P be2413644ddc406da9fd02b7bd712f09: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:21.565662 31702 ts_tablet_manager.cc:1367] T 5ceb582179914d678d162c4320c3eae3 P 6716d6d1cba44ec1bcc584707531fba1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:58:21.566202 31702 raft_consensus.cc:348] T 5ceb582179914d678d162c4320c3eae3 P 6716d6d1cba44ec1bcc584707531fba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6716d6d1cba44ec1bcc584707531fba1" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 44715 } }
I20240502 07:58:21.566309 31702 raft_consensus.cc:374] T 5ceb582179914d678d162c4320c3eae3 P 6716d6d1cba44ec1bcc584707531fba1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:21.566345 31702 raft_consensus.cc:729] T 5ceb582179914d678d162c4320c3eae3 P 6716d6d1cba44ec1bcc584707531fba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6716d6d1cba44ec1bcc584707531fba1, State: Initialized, Role: FOLLOWER
I20240502 07:58:21.566524 31702 consensus_queue.cc:260] T 5ceb582179914d678d162c4320c3eae3 P 6716d6d1cba44ec1bcc584707531fba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6716d6d1cba44ec1bcc584707531fba1" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 44715 } }
I20240502 07:58:21.566594 31701 raft_consensus.cc:504] T 9a17a897c6f744f09e7c50101b773a40 P 77ec61f2c97b4db9983578fe9d587dba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77ec61f2c97b4db9983578fe9d587dba" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42199 } }
I20240502 07:58:21.566776 31705 tablet_bootstrap.cc:492] T f0b23d51073649cea865478bb091bdf2 P be2413644ddc406da9fd02b7bd712f09: No bootstrap required, opened a new log
I20240502 07:58:21.566601 31702 raft_consensus.cc:388] T 5ceb582179914d678d162c4320c3eae3 P 6716d6d1cba44ec1bcc584707531fba1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:58:21.566893 31702 raft_consensus.cc:482] T 5ceb582179914d678d162c4320c3eae3 P 6716d6d1cba44ec1bcc584707531fba1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:58:21.566893 31705 ts_tablet_manager.cc:1367] T f0b23d51073649cea865478bb091bdf2 P be2413644ddc406da9fd02b7bd712f09: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:21.566977 31702 raft_consensus.cc:3037] T 5ceb582179914d678d162c4320c3eae3 P 6716d6d1cba44ec1bcc584707531fba1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:21.567423 31705 raft_consensus.cc:348] T f0b23d51073649cea865478bb091bdf2 P be2413644ddc406da9fd02b7bd712f09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be2413644ddc406da9fd02b7bd712f09" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 43521 } }
I20240502 07:58:21.567517 31705 raft_consensus.cc:374] T f0b23d51073649cea865478bb091bdf2 P be2413644ddc406da9fd02b7bd712f09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:21.567559 31705 raft_consensus.cc:729] T f0b23d51073649cea865478bb091bdf2 P be2413644ddc406da9fd02b7bd712f09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be2413644ddc406da9fd02b7bd712f09, State: Initialized, Role: FOLLOWER
I20240502 07:58:21.567709 31705 consensus_queue.cc:260] T f0b23d51073649cea865478bb091bdf2 P be2413644ddc406da9fd02b7bd712f09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be2413644ddc406da9fd02b7bd712f09" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 43521 } }
I20240502 07:58:21.567898 31705 raft_consensus.cc:388] T f0b23d51073649cea865478bb091bdf2 P be2413644ddc406da9fd02b7bd712f09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:58:21.567966 31705 raft_consensus.cc:482] T f0b23d51073649cea865478bb091bdf2 P be2413644ddc406da9fd02b7bd712f09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:58:21.566763 31701 leader_election.cc:302] T 9a17a897c6f744f09e7c50101b773a40 P 77ec61f2c97b4db9983578fe9d587dba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77ec61f2c97b4db9983578fe9d587dba; no voters:
I20240502 07:58:21.568064 31705 raft_consensus.cc:3037] T f0b23d51073649cea865478bb091bdf2 P be2413644ddc406da9fd02b7bd712f09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:21.568418 31701 leader_election.cc:288] T 9a17a897c6f744f09e7c50101b773a40 P 77ec61f2c97b4db9983578fe9d587dba [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:58:21.568708 31708 raft_consensus.cc:2781] T 9a17a897c6f744f09e7c50101b773a40 P 77ec61f2c97b4db9983578fe9d587dba [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:21.568766 31701 ts_tablet_manager.cc:1398] T 9a17a897c6f744f09e7c50101b773a40 P 77ec61f2c97b4db9983578fe9d587dba: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 07:58:21.568074 31702 raft_consensus.cc:504] T 5ceb582179914d678d162c4320c3eae3 P 6716d6d1cba44ec1bcc584707531fba1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6716d6d1cba44ec1bcc584707531fba1" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 44715 } }
I20240502 07:58:21.568933 31708 raft_consensus.cc:686] T 9a17a897c6f744f09e7c50101b773a40 P 77ec61f2c97b4db9983578fe9d587dba [term 1 LEADER]: Becoming Leader. State: Replica: 77ec61f2c97b4db9983578fe9d587dba, State: Running, Role: LEADER
I20240502 07:58:21.569101 31702 leader_election.cc:302] T 5ceb582179914d678d162c4320c3eae3 P 6716d6d1cba44ec1bcc584707531fba1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6716d6d1cba44ec1bcc584707531fba1; no voters:
I20240502 07:58:21.569168 31708 consensus_queue.cc:237] T 9a17a897c6f744f09e7c50101b773a40 P 77ec61f2c97b4db9983578fe9d587dba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77ec61f2c97b4db9983578fe9d587dba" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42199 } }
I20240502 07:58:21.569245 31705 raft_consensus.cc:504] T f0b23d51073649cea865478bb091bdf2 P be2413644ddc406da9fd02b7bd712f09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be2413644ddc406da9fd02b7bd712f09" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 43521 } }
I20240502 07:58:21.569413 31705 leader_election.cc:302] T f0b23d51073649cea865478bb091bdf2 P be2413644ddc406da9fd02b7bd712f09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be2413644ddc406da9fd02b7bd712f09; no voters:
I20240502 07:58:21.569603 31705 leader_election.cc:288] T f0b23d51073649cea865478bb091bdf2 P be2413644ddc406da9fd02b7bd712f09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:58:21.569689 31712 raft_consensus.cc:2781] T f0b23d51073649cea865478bb091bdf2 P be2413644ddc406da9fd02b7bd712f09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:21.569943 31712 raft_consensus.cc:686] T f0b23d51073649cea865478bb091bdf2 P be2413644ddc406da9fd02b7bd712f09 [term 1 LEADER]: Becoming Leader. State: Replica: be2413644ddc406da9fd02b7bd712f09, State: Running, Role: LEADER
I20240502 07:58:21.569963 31705 ts_tablet_manager.cc:1398] T f0b23d51073649cea865478bb091bdf2 P be2413644ddc406da9fd02b7bd712f09: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:21.570155 31712 consensus_queue.cc:237] T f0b23d51073649cea865478bb091bdf2 P be2413644ddc406da9fd02b7bd712f09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be2413644ddc406da9fd02b7bd712f09" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 43521 } }
I20240502 07:58:21.571291 31279 catalog_manager.cc:5484] T 9a17a897c6f744f09e7c50101b773a40 P 77ec61f2c97b4db9983578fe9d587dba reported cstate change: term changed from 0 to 1, leader changed from <none> to 77ec61f2c97b4db9983578fe9d587dba (127.29.106.129). New cstate: current_term: 1 leader_uuid: "77ec61f2c97b4db9983578fe9d587dba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77ec61f2c97b4db9983578fe9d587dba" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42199 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:21.569326 31702 leader_election.cc:288] T 5ceb582179914d678d162c4320c3eae3 P 6716d6d1cba44ec1bcc584707531fba1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:58:21.573060 31702 ts_tablet_manager.cc:1398] T 5ceb582179914d678d162c4320c3eae3 P 6716d6d1cba44ec1bcc584707531fba1: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240502 07:58:21.573184 31278 catalog_manager.cc:5484] T f0b23d51073649cea865478bb091bdf2 P be2413644ddc406da9fd02b7bd712f09 reported cstate change: term changed from 0 to 1, leader changed from <none> to be2413644ddc406da9fd02b7bd712f09 (127.29.106.131). New cstate: current_term: 1 leader_uuid: "be2413644ddc406da9fd02b7bd712f09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be2413644ddc406da9fd02b7bd712f09" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 43521 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:21.573490 31710 raft_consensus.cc:2781] T 5ceb582179914d678d162c4320c3eae3 P 6716d6d1cba44ec1bcc584707531fba1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:21.573683 31710 raft_consensus.cc:686] T 5ceb582179914d678d162c4320c3eae3 P 6716d6d1cba44ec1bcc584707531fba1 [term 1 LEADER]: Becoming Leader. State: Replica: 6716d6d1cba44ec1bcc584707531fba1, State: Running, Role: LEADER
I20240502 07:58:21.573809 31710 consensus_queue.cc:237] T 5ceb582179914d678d162c4320c3eae3 P 6716d6d1cba44ec1bcc584707531fba1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6716d6d1cba44ec1bcc584707531fba1" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 44715 } }
I20240502 07:58:21.577093 31277 catalog_manager.cc:5484] T 5ceb582179914d678d162c4320c3eae3 P 6716d6d1cba44ec1bcc584707531fba1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6716d6d1cba44ec1bcc584707531fba1 (127.29.106.130). New cstate: current_term: 1 leader_uuid: "6716d6d1cba44ec1bcc584707531fba1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6716d6d1cba44ec1bcc584707531fba1" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 44715 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:21.616236 30122 master.cc:537] Master@127.29.106.189:40887 shutting down...
I20240502 07:58:21.620857 30122 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6 [term 1 LEADER]: Raft consensus shutting down.
I20240502 07:58:21.621316 30122 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:21.621428 30122 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1298d969e51c4856884d33ce860464b6: stopping tablet replica
I20240502 07:58:21.638854 30122 master.cc:552] Master@127.29.106.189:40887 shutdown complete.
W20240502 07:58:22.587833 31498 heartbeater.cc:644] Failed to heartbeat to 127.29.106.189:40887 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.29.106.189:40887: connect: Connection refused (error 111)
I20240502 07:58:23.089957 31726 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1298d969e51c4856884d33ce860464b6)
I20240502 07:58:23.090266 31726 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8e51c20fd46d18c090295db4fca94" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 } } peers { permanent_uuid: "1298d969e51c4856884d33ce860464b6" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 40887 } } peers { permanent_uuid: "c8d8e32856424b588814b81dff25aa25" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } }
I20240502 07:58:23.090723 31726 leader_election.cc:288] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1298d969e51c4856884d33ce860464b6 (127.29.106.189:40887), c8d8e32856424b588814b81dff25aa25 (127.29.106.188:38279)
I20240502 07:58:23.091552 31355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e3d8e51c20fd46d18c090295db4fca94" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "c8d8e32856424b588814b81dff25aa25" is_pre_election: true
W20240502 07:58:23.092391 31197 leader_election.cc:334] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1298d969e51c4856884d33ce860464b6 (127.29.106.189:40887): Network error: Client connection negotiation failed: client connection to 127.29.106.189:40887: connect: Connection refused (error 111)
I20240502 07:58:23.092468 31197 leader_election.cc:302] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3d8e51c20fd46d18c090295db4fca94; no voters: 1298d969e51c4856884d33ce860464b6, c8d8e32856424b588814b81dff25aa25
I20240502 07:58:23.092609 31726 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 07:58:23.099419 31729 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1298d969e51c4856884d33ce860464b6)
I20240502 07:58:23.099622 31729 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8e51c20fd46d18c090295db4fca94" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 } } peers { permanent_uuid: "1298d969e51c4856884d33ce860464b6" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 40887 } } peers { permanent_uuid: "c8d8e32856424b588814b81dff25aa25" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } }
I20240502 07:58:23.100320 31729 leader_election.cc:288] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e3d8e51c20fd46d18c090295db4fca94 (127.29.106.190:43859), 1298d969e51c4856884d33ce860464b6 (127.29.106.189:40887)
I20240502 07:58:23.100646 31223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8d8e32856424b588814b81dff25aa25" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "e3d8e51c20fd46d18c090295db4fca94" is_pre_election: true
I20240502 07:58:23.100811 31223 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d8e32856424b588814b81dff25aa25 in term 1.
I20240502 07:58:23.101166 31329 leader_election.cc:302] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8d8e32856424b588814b81dff25aa25, e3d8e51c20fd46d18c090295db4fca94; no voters:
I20240502 07:58:23.101400 31729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 07:58:23.101460 31729 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1298d969e51c4856884d33ce860464b6)
I20240502 07:58:23.101511 31729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [term 1 FOLLOWER]: Advancing to term 2
W20240502 07:58:23.101717 31328 leader_election.cc:334] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1298d969e51c4856884d33ce860464b6 (127.29.106.189:40887): Network error: Client connection negotiation failed: client connection to 127.29.106.189:40887: connect: Connection refused (error 111)
I20240502 07:58:23.102982 31729 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8e51c20fd46d18c090295db4fca94" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 } } peers { permanent_uuid: "1298d969e51c4856884d33ce860464b6" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 40887 } } peers { permanent_uuid: "c8d8e32856424b588814b81dff25aa25" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } }
I20240502 07:58:23.103348 31729 leader_election.cc:288] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [CANDIDATE]: Term 2 election: Requested vote from peers e3d8e51c20fd46d18c090295db4fca94 (127.29.106.190:43859), 1298d969e51c4856884d33ce860464b6 (127.29.106.189:40887)
I20240502 07:58:23.103729 31223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8d8e32856424b588814b81dff25aa25" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "e3d8e51c20fd46d18c090295db4fca94"
I20240502 07:58:23.103827 31223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94 [term 1 FOLLOWER]: Advancing to term 2
W20240502 07:58:23.104439 31328 leader_election.cc:334] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1298d969e51c4856884d33ce860464b6 (127.29.106.189:40887): Network error: Client connection negotiation failed: client connection to 127.29.106.189:40887: connect: Connection refused (error 111)
I20240502 07:58:23.104873 31223 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d8e32856424b588814b81dff25aa25 in term 2.
I20240502 07:58:23.105242 31329 leader_election.cc:302] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c8d8e32856424b588814b81dff25aa25, e3d8e51c20fd46d18c090295db4fca94; no voters: 1298d969e51c4856884d33ce860464b6
I20240502 07:58:23.105440 31729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:58:23.105690 31729 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [term 2 LEADER]: Becoming Leader. State: Replica: c8d8e32856424b588814b81dff25aa25, State: Running, Role: LEADER
I20240502 07:58:23.105849 31729 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8e51c20fd46d18c090295db4fca94" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 } } peers { permanent_uuid: "1298d969e51c4856884d33ce860464b6" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 40887 } } peers { permanent_uuid: "c8d8e32856424b588814b81dff25aa25" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } }
I20240502 07:58:23.106568 31732 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8d8e32856424b588814b81dff25aa25. Latest consensus state: current_term: 2 leader_uuid: "c8d8e32856424b588814b81dff25aa25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8e51c20fd46d18c090295db4fca94" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 } } peers { permanent_uuid: "1298d969e51c4856884d33ce860464b6" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 40887 } } peers { permanent_uuid: "c8d8e32856424b588814b81dff25aa25" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } } }
I20240502 07:58:23.106690 31732 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [sys.catalog]: This master's current role is: LEADER
I20240502 07:58:23.107062 31734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 07:58:23.108489 31734 catalog_manager.cc:669] Loaded metadata for table test-workload [id=eb42958ed31848a0845250bbee3ce6e8]
I20240502 07:58:23.109298 31734 tablet_loader.cc:96] loaded metadata for tablet 5ceb582179914d678d162c4320c3eae3 (table test-workload [id=eb42958ed31848a0845250bbee3ce6e8])
I20240502 07:58:23.109508 31734 tablet_loader.cc:96] loaded metadata for tablet 9a17a897c6f744f09e7c50101b773a40 (table test-workload [id=eb42958ed31848a0845250bbee3ce6e8])
I20240502 07:58:23.109689 31734 tablet_loader.cc:96] loaded metadata for tablet f0b23d51073649cea865478bb091bdf2 (table test-workload [id=eb42958ed31848a0845250bbee3ce6e8])
I20240502 07:58:23.109786 31734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 07:58:23.110193 31734 catalog_manager.cc:1259] Loaded cluster ID: c3223e60d99445a78d005007160aceb2
I20240502 07:58:23.110241 31734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 07:58:23.110806 31734 catalog_manager.cc:1504] Loading token signing keys...
I20240502 07:58:23.111215 31734 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25: Loaded TSK: 0
I20240502 07:58:23.111755 31734 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 07:58:23.540251 31223 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94 [term 2 FOLLOWER]: Refusing update from remote peer c8d8e32856424b588814b81dff25aa25: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 07:58:23.540937 31732 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3d8e51c20fd46d18c090295db4fca94" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:58:23.542376 31726 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8d8e32856424b588814b81dff25aa25. Latest consensus state: current_term: 2 leader_uuid: "c8d8e32856424b588814b81dff25aa25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8e51c20fd46d18c090295db4fca94" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 } } peers { permanent_uuid: "1298d969e51c4856884d33ce860464b6" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 40887 } } peers { permanent_uuid: "c8d8e32856424b588814b81dff25aa25" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } } }
I20240502 07:58:23.542513 31726 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 07:58:23.543802 31729 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "c8d8e32856424b588814b81dff25aa25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8e51c20fd46d18c090295db4fca94" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 } } peers { permanent_uuid: "1298d969e51c4856884d33ce860464b6" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 40887 } } peers { permanent_uuid: "c8d8e32856424b588814b81dff25aa25" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } } }
I20240502 07:58:23.544059 31729 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [sys.catalog]: This master's current role is: LEADER
I20240502 07:58:23.544036 31726 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "c8d8e32856424b588814b81dff25aa25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d8e51c20fd46d18c090295db4fca94" member_type: VOTER last_known_addr { host: "127.29.106.190" port: 43859 } } peers { permanent_uuid: "1298d969e51c4856884d33ce860464b6" member_type: VOTER last_known_addr { host: "127.29.106.189" port: 40887 } } peers { permanent_uuid: "c8d8e32856424b588814b81dff25aa25" member_type: VOTER last_known_addr { host: "127.29.106.188" port: 38279 } } }
I20240502 07:58:23.544170 31726 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94 [sys.catalog]: This master's current role is: FOLLOWER
W20240502 07:58:23.544874 31328 consensus_peers.cc:485] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 -> Peer 1298d969e51c4856884d33ce860464b6 (127.29.106.189:40887): Couldn't send request to peer 1298d969e51c4856884d33ce860464b6. Status: Network error: Client connection negotiation failed: client connection to 127.29.106.189:40887: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 07:58:23.973803 31328 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.106.189:40887: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240502 07:58:24.579917 31496 heartbeater.cc:503] Master 127.29.106.188:38279 was elected leader, sending a full tablet report...
I20240502 07:58:24.584463 31575 heartbeater.cc:503] Master 127.29.106.188:38279 was elected leader, sending a full tablet report...
I20240502 07:58:24.585026 31654 heartbeater.cc:503] Master 127.29.106.188:38279 was elected leader, sending a full tablet report...
I20240502 07:58:25.091346 31409 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:58:25.091480 31409 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:58:25.180578 30122 tablet_server.cc:178] TabletServer@127.29.106.129:0 shutting down...
I20240502 07:58:25.186157 30122 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 07:58:25.186484 30122 tablet_replica.cc:330] T 9a17a897c6f744f09e7c50101b773a40 P 77ec61f2c97b4db9983578fe9d587dba: stopping tablet replica
I20240502 07:58:25.186633 30122 raft_consensus.cc:2229] T 9a17a897c6f744f09e7c50101b773a40 P 77ec61f2c97b4db9983578fe9d587dba [term 1 LEADER]: Raft consensus shutting down.
I20240502 07:58:25.186813 30122 raft_consensus.cc:2258] T 9a17a897c6f744f09e7c50101b773a40 P 77ec61f2c97b4db9983578fe9d587dba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:25.200101 30122 tablet_server.cc:194] TabletServer@127.29.106.129:0 shutdown complete.
I20240502 07:58:25.202848 30122 tablet_server.cc:178] TabletServer@127.29.106.130:0 shutting down...
I20240502 07:58:25.210408 30122 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 07:58:25.210719 30122 tablet_replica.cc:330] T 5ceb582179914d678d162c4320c3eae3 P 6716d6d1cba44ec1bcc584707531fba1: stopping tablet replica
I20240502 07:58:25.210872 30122 raft_consensus.cc:2229] T 5ceb582179914d678d162c4320c3eae3 P 6716d6d1cba44ec1bcc584707531fba1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 07:58:25.211048 30122 raft_consensus.cc:2258] T 5ceb582179914d678d162c4320c3eae3 P 6716d6d1cba44ec1bcc584707531fba1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:25.224603 30122 tablet_server.cc:194] TabletServer@127.29.106.130:0 shutdown complete.
I20240502 07:58:25.227313 30122 tablet_server.cc:178] TabletServer@127.29.106.131:0 shutting down...
I20240502 07:58:25.233378 30122 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 07:58:25.233631 30122 tablet_replica.cc:330] T f0b23d51073649cea865478bb091bdf2 P be2413644ddc406da9fd02b7bd712f09: stopping tablet replica
I20240502 07:58:25.233788 30122 raft_consensus.cc:2229] T f0b23d51073649cea865478bb091bdf2 P be2413644ddc406da9fd02b7bd712f09 [term 1 LEADER]: Raft consensus shutting down.
I20240502 07:58:25.233938 30122 raft_consensus.cc:2258] T f0b23d51073649cea865478bb091bdf2 P be2413644ddc406da9fd02b7bd712f09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:25.248636 30122 tablet_server.cc:194] TabletServer@127.29.106.131:0 shutdown complete.
I20240502 07:58:25.251453 30122 master.cc:537] Master@127.29.106.190:43859 shutting down...
I20240502 07:58:25.255712 30122 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:25.255940 30122 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:25.256017 30122 tablet_replica.cc:330] T 00000000000000000000000000000000 P e3d8e51c20fd46d18c090295db4fca94: stopping tablet replica
I20240502 07:58:25.269884 30122 master.cc:552] Master@127.29.106.190:43859 shutdown complete.
I20240502 07:58:25.273519 30122 master.cc:537] Master@127.29.106.188:38279 shutting down...
I20240502 07:58:25.278316 30122 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [term 2 LEADER]: Raft consensus shutting down.
I20240502 07:58:25.278689 30122 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:25.278766 30122 tablet_replica.cc:330] T 00000000000000000000000000000000 P c8d8e32856424b588814b81dff25aa25: stopping tablet replica
I20240502 07:58:25.292068 30122 master.cc:552] Master@127.29.106.188:38279 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (6346 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20240502 07:58:25.301402 30122 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.106.190:40157
I20240502 07:58:25.301805 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:25.305215 31750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:25.305836 30122 server_base.cc:1034] running on GCE node
W20240502 07:58:25.306300 31752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:25.307313 31749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:25.307598 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:58:25.307654 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:58:25.307680 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636705307679 us; error 0 us; skew 500 ppm
I20240502 07:58:25.314168 30122 webserver.cc:460] Webserver started at http://127.29.106.190:46063/ using document root <none> and password file <none>
I20240502 07:58:25.314405 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:25.314469 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:25.314575 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:25.315069 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636688849691-30122-0/minicluster-data/master-0-root/instance:
uuid: "3dd356a7497f48efb4ba5523cf97dc50"
format_stamp: "Formatted at 2024-05-02 07:58:25 on dist-test-slave-m6l2"
I20240502 07:58:25.316898 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20240502 07:58:25.318065 31757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:25.318571 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:25.318676 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636688849691-30122-0/minicluster-data/master-0-root
uuid: "3dd356a7497f48efb4ba5523cf97dc50"
format_stamp: "Formatted at 2024-05-02 07:58:25 on dist-test-slave-m6l2"
I20240502 07:58:25.318765 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636688849691-30122-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636688849691-30122-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636688849691-30122-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:25.334528 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:25.335048 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:25.340045 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.190:40157
I20240502 07:58:25.343253 31809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.190:40157 every 8 connection(s)
I20240502 07:58:25.343781 31810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:25.345993 31810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dd356a7497f48efb4ba5523cf97dc50: Bootstrap starting.
I20240502 07:58:25.346784 31810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3dd356a7497f48efb4ba5523cf97dc50: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:25.348090 31810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dd356a7497f48efb4ba5523cf97dc50: No bootstrap required, opened a new log
I20240502 07:58:25.348531 31810 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3dd356a7497f48efb4ba5523cf97dc50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd356a7497f48efb4ba5523cf97dc50" member_type: VOTER }
I20240502 07:58:25.348625 31810 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3dd356a7497f48efb4ba5523cf97dc50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:25.348647 31810 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3dd356a7497f48efb4ba5523cf97dc50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dd356a7497f48efb4ba5523cf97dc50, State: Initialized, Role: FOLLOWER
I20240502 07:58:25.348789 31810 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3dd356a7497f48efb4ba5523cf97dc50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd356a7497f48efb4ba5523cf97dc50" member_type: VOTER }
I20240502 07:58:25.348867 31810 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3dd356a7497f48efb4ba5523cf97dc50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:58:25.348917 31810 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3dd356a7497f48efb4ba5523cf97dc50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:58:25.348971 31810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3dd356a7497f48efb4ba5523cf97dc50 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:25.349696 31810 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3dd356a7497f48efb4ba5523cf97dc50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd356a7497f48efb4ba5523cf97dc50" member_type: VOTER }
I20240502 07:58:25.349817 31810 leader_election.cc:302] T 00000000000000000000000000000000 P 3dd356a7497f48efb4ba5523cf97dc50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3dd356a7497f48efb4ba5523cf97dc50; no voters:
I20240502 07:58:25.350037 31810 leader_election.cc:288] T 00000000000000000000000000000000 P 3dd356a7497f48efb4ba5523cf97dc50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:58:25.350244 31813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3dd356a7497f48efb4ba5523cf97dc50 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:25.350474 31810 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3dd356a7497f48efb4ba5523cf97dc50 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 07:58:25.350525 31813 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3dd356a7497f48efb4ba5523cf97dc50 [term 1 LEADER]: Becoming Leader. State: Replica: 3dd356a7497f48efb4ba5523cf97dc50, State: Running, Role: LEADER
I20240502 07:58:25.350791 31813 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3dd356a7497f48efb4ba5523cf97dc50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd356a7497f48efb4ba5523cf97dc50" member_type: VOTER }
I20240502 07:58:25.351406 31815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dd356a7497f48efb4ba5523cf97dc50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3dd356a7497f48efb4ba5523cf97dc50. Latest consensus state: current_term: 1 leader_uuid: "3dd356a7497f48efb4ba5523cf97dc50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd356a7497f48efb4ba5523cf97dc50" member_type: VOTER } }
I20240502 07:58:25.351521 31815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dd356a7497f48efb4ba5523cf97dc50 [sys.catalog]: This master's current role is: LEADER
I20240502 07:58:25.351831 31822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 07:58:25.351984 31814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dd356a7497f48efb4ba5523cf97dc50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3dd356a7497f48efb4ba5523cf97dc50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd356a7497f48efb4ba5523cf97dc50" member_type: VOTER } }
I20240502 07:58:25.352195 31814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dd356a7497f48efb4ba5523cf97dc50 [sys.catalog]: This master's current role is: LEADER
I20240502 07:58:25.352931 31822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 07:58:25.353206 30122 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 07:58:25.354950 31822 catalog_manager.cc:1347] Generated new cluster ID: 55cbfce76db54c6f9631bb6607623580
I20240502 07:58:25.355026 31822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 07:58:25.363745 31822 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 07:58:25.364409 31822 catalog_manager.cc:1504] Loading token signing keys...
I20240502 07:58:25.371274 31822 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3dd356a7497f48efb4ba5523cf97dc50: Generated new TSK 0
I20240502 07:58:25.371523 31822 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 07:58:25.386061 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:25.389313 31834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:25.389850 31832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:25.390595 31831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:25.391448 30122 server_base.cc:1034] running on GCE node
I20240502 07:58:25.391639 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:58:25.391680 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:58:25.391697 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636705391697 us; error 0 us; skew 500 ppm
I20240502 07:58:25.393003 30122 webserver.cc:460] Webserver started at http://127.29.106.129:43741/ using document root <none> and password file <none>
I20240502 07:58:25.393190 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:25.393246 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:25.393328 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:25.393817 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636688849691-30122-0/minicluster-data/ts-0-root/instance:
uuid: "e9c07145e2d845b680ba06c14c6e0e5c"
format_stamp: "Formatted at 2024-05-02 07:58:25 on dist-test-slave-m6l2"
I20240502 07:58:25.395634 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20240502 07:58:25.396915 31839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:25.397291 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:25.397397 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636688849691-30122-0/minicluster-data/ts-0-root
uuid: "e9c07145e2d845b680ba06c14c6e0e5c"
format_stamp: "Formatted at 2024-05-02 07:58:25 on dist-test-slave-m6l2"
I20240502 07:58:25.397480 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636688849691-30122-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636688849691-30122-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636688849691-30122-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:25.409722 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:25.410192 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:25.410531 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:25.411059 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:25.411106 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:25.411140 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:25.411156 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:25.416086 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.129:42333
I20240502 07:58:25.416158 31902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.129:42333 every 8 connection(s)
I20240502 07:58:25.417025 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:25.420083 31907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:25.420641 31910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:25.421663 31908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:25.422214 31903 heartbeater.cc:347] Connected to a master server at 127.29.106.190:40157
I20240502 07:58:25.422341 30122 server_base.cc:1034] running on GCE node
I20240502 07:58:25.422358 31903 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:25.422545 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:58:25.422600 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:58:25.422623 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636705422623 us; error 0 us; skew 500 ppm
I20240502 07:58:25.422708 31903 heartbeater.cc:511] Master 127.29.106.190:40157 requested a full tablet report, sending...
I20240502 07:58:25.423697 31774 ts_manager.cc:194] Registered new tserver with Master: e9c07145e2d845b680ba06c14c6e0e5c (127.29.106.129:42333)
I20240502 07:58:25.423794 30122 webserver.cc:460] Webserver started at http://127.29.106.130:43185/ using document root <none> and password file <none>
I20240502 07:58:25.423977 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:25.424033 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:25.424098 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:25.424578 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636688849691-30122-0/minicluster-data/ts-1-root/instance:
uuid: "f9e57376395e46fb8e35ed8f1b613695"
format_stamp: "Formatted at 2024-05-02 07:58:25 on dist-test-slave-m6l2"
I20240502 07:58:25.424696 31774 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35724
I20240502 07:58:25.426426 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:25.427469 31915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:25.427814 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:25.427896 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636688849691-30122-0/minicluster-data/ts-1-root
uuid: "f9e57376395e46fb8e35ed8f1b613695"
format_stamp: "Formatted at 2024-05-02 07:58:25 on dist-test-slave-m6l2"
I20240502 07:58:25.427979 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636688849691-30122-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636688849691-30122-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636688849691-30122-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:25.434880 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:25.435263 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:25.435542 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:25.436053 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:25.436094 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:25.436136 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:25.436169 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:25.440997 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.130:39621
I20240502 07:58:25.441056 31978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.130:39621 every 8 connection(s)
I20240502 07:58:25.441905 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:25.444908 31982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:25.445749 31983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:25.446497 31985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:25.446650 31979 heartbeater.cc:347] Connected to a master server at 127.29.106.190:40157
I20240502 07:58:25.446753 31979 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:25.446985 31979 heartbeater.cc:511] Master 127.29.106.190:40157 requested a full tablet report, sending...
I20240502 07:58:25.447059 30122 server_base.cc:1034] running on GCE node
I20240502 07:58:25.447273 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:58:25.447319 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:58:25.447345 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636705447342 us; error 0 us; skew 500 ppm
I20240502 07:58:25.447692 31774 ts_manager.cc:194] Registered new tserver with Master: f9e57376395e46fb8e35ed8f1b613695 (127.29.106.130:39621)
I20240502 07:58:25.448565 30122 webserver.cc:460] Webserver started at http://127.29.106.131:33303/ using document root <none> and password file <none>
I20240502 07:58:25.448657 31774 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35728
I20240502 07:58:25.448742 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:25.448808 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:25.448891 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:25.449357 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636688849691-30122-0/minicluster-data/ts-2-root/instance:
uuid: "7feef95690fe4b0bb736a884af9569e6"
format_stamp: "Formatted at 2024-05-02 07:58:25 on dist-test-slave-m6l2"
I20240502 07:58:25.450949 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 07:58:25.451893 31990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:25.452117 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:25.452190 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636688849691-30122-0/minicluster-data/ts-2-root
uuid: "7feef95690fe4b0bb736a884af9569e6"
format_stamp: "Formatted at 2024-05-02 07:58:25 on dist-test-slave-m6l2"
I20240502 07:58:25.452311 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636688849691-30122-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636688849691-30122-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636688849691-30122-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:25.461658 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:25.462050 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:25.462368 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:25.462834 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:25.462881 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:25.462930 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:25.462961 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:25.467360 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.131:41637
I20240502 07:58:25.467417 32053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.131:41637 every 8 connection(s)
I20240502 07:58:25.471750 32054 heartbeater.cc:347] Connected to a master server at 127.29.106.190:40157
I20240502 07:58:25.471855 32054 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:25.472060 32054 heartbeater.cc:511] Master 127.29.106.190:40157 requested a full tablet report, sending...
I20240502 07:58:25.472681 31774 ts_manager.cc:194] Registered new tserver with Master: 7feef95690fe4b0bb736a884af9569e6 (127.29.106.131:41637)
I20240502 07:58:25.473177 31774 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35732
I20240502 07:58:25.473361 30122 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005578473s
I20240502 07:58:26.425902 31903 heartbeater.cc:503] Master 127.29.106.190:40157 was elected leader, sending a full tablet report...
I20240502 07:58:26.449743 31979 heartbeater.cc:503] Master 127.29.106.190:40157 was elected leader, sending a full tablet report...
I20240502 07:58:26.474184 32054 heartbeater.cc:503] Master 127.29.106.190:40157 was elected leader, sending a full tablet report...
I20240502 07:58:26.499730 30122 test_util.cc:268] Using random seed: -137409393
I20240502 07:58:26.509189 31774 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43028:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 07:58:26.509712 31774 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 07:58:26.522308 31867 tablet_service.cc:1467] Processing CreateTablet for tablet 862e4df4c8004aa0973a06875b587e5f (DEFAULT_TABLE table=test-workload [id=9927fd31327440198002a2789860edcc]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 07:58:26.522329 32018 tablet_service.cc:1467] Processing CreateTablet for tablet 862e4df4c8004aa0973a06875b587e5f (DEFAULT_TABLE table=test-workload [id=9927fd31327440198002a2789860edcc]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 07:58:26.522724 31867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 862e4df4c8004aa0973a06875b587e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:26.522329 32017 tablet_service.cc:1467] Processing CreateTablet for tablet 7343c21e7dfe47a2bfcc793de456d971 (DEFAULT_TABLE table=test-workload [id=9927fd31327440198002a2789860edcc]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 07:58:26.523052 31866 tablet_service.cc:1467] Processing CreateTablet for tablet 7343c21e7dfe47a2bfcc793de456d971 (DEFAULT_TABLE table=test-workload [id=9927fd31327440198002a2789860edcc]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 07:58:26.523355 31943 tablet_service.cc:1467] Processing CreateTablet for tablet 862e4df4c8004aa0973a06875b587e5f (DEFAULT_TABLE table=test-workload [id=9927fd31327440198002a2789860edcc]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 07:58:26.523365 31866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7343c21e7dfe47a2bfcc793de456d971. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:26.523622 31943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 862e4df4c8004aa0973a06875b587e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:26.522750 32018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 862e4df4c8004aa0973a06875b587e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:26.526957 32075 tablet_bootstrap.cc:492] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6: Bootstrap starting.
I20240502 07:58:26.527556 31942 tablet_service.cc:1467] Processing CreateTablet for tablet 7343c21e7dfe47a2bfcc793de456d971 (DEFAULT_TABLE table=test-workload [id=9927fd31327440198002a2789860edcc]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 07:58:26.527897 31942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7343c21e7dfe47a2bfcc793de456d971. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:26.528337 32075 tablet_bootstrap.cc:654] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:26.528630 32076 tablet_bootstrap.cc:492] T 862e4df4c8004aa0973a06875b587e5f P f9e57376395e46fb8e35ed8f1b613695: Bootstrap starting.
I20240502 07:58:26.529613 32075 tablet_bootstrap.cc:492] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6: No bootstrap required, opened a new log
I20240502 07:58:26.529651 32074 tablet_bootstrap.cc:492] T 862e4df4c8004aa0973a06875b587e5f P e9c07145e2d845b680ba06c14c6e0e5c: Bootstrap starting.
I20240502 07:58:26.529713 32075 ts_tablet_manager.cc:1367] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 07:58:26.530242 32075 raft_consensus.cc:348] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } } peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } }
I20240502 07:58:26.530346 32076 tablet_bootstrap.cc:654] T 862e4df4c8004aa0973a06875b587e5f P f9e57376395e46fb8e35ed8f1b613695: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:26.530365 32075 raft_consensus.cc:374] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:26.530400 32075 raft_consensus.cc:729] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7feef95690fe4b0bb736a884af9569e6, State: Initialized, Role: FOLLOWER
I20240502 07:58:26.530537 32075 consensus_queue.cc:260] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } } peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } }
I20240502 07:58:26.530642 32017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7343c21e7dfe47a2bfcc793de456d971. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:26.530766 32074 tablet_bootstrap.cc:654] T 862e4df4c8004aa0973a06875b587e5f P e9c07145e2d845b680ba06c14c6e0e5c: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:26.530943 32075 ts_tablet_manager.cc:1398] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:26.531913 32076 tablet_bootstrap.cc:492] T 862e4df4c8004aa0973a06875b587e5f P f9e57376395e46fb8e35ed8f1b613695: No bootstrap required, opened a new log
I20240502 07:58:26.532088 32074 tablet_bootstrap.cc:492] T 862e4df4c8004aa0973a06875b587e5f P e9c07145e2d845b680ba06c14c6e0e5c: No bootstrap required, opened a new log
I20240502 07:58:26.532101 32076 ts_tablet_manager.cc:1367] T 862e4df4c8004aa0973a06875b587e5f P f9e57376395e46fb8e35ed8f1b613695: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:58:26.532339 32074 ts_tablet_manager.cc:1367] T 862e4df4c8004aa0973a06875b587e5f P e9c07145e2d845b680ba06c14c6e0e5c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:26.533097 32074 raft_consensus.cc:348] T 862e4df4c8004aa0973a06875b587e5f P e9c07145e2d845b680ba06c14c6e0e5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } } peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } }
I20240502 07:58:26.533247 32074 raft_consensus.cc:374] T 862e4df4c8004aa0973a06875b587e5f P e9c07145e2d845b680ba06c14c6e0e5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:26.533300 32074 raft_consensus.cc:729] T 862e4df4c8004aa0973a06875b587e5f P e9c07145e2d845b680ba06c14c6e0e5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c07145e2d845b680ba06c14c6e0e5c, State: Initialized, Role: FOLLOWER
I20240502 07:58:26.533342 32075 tablet_bootstrap.cc:492] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6: Bootstrap starting.
I20240502 07:58:26.533434 32074 consensus_queue.cc:260] T 862e4df4c8004aa0973a06875b587e5f P e9c07145e2d845b680ba06c14c6e0e5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } } peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } }
I20240502 07:58:26.533824 32074 ts_tablet_manager.cc:1398] T 862e4df4c8004aa0973a06875b587e5f P e9c07145e2d845b680ba06c14c6e0e5c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:26.534013 32074 tablet_bootstrap.cc:492] T 7343c21e7dfe47a2bfcc793de456d971 P e9c07145e2d845b680ba06c14c6e0e5c: Bootstrap starting.
I20240502 07:58:26.534370 32075 tablet_bootstrap.cc:654] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:26.535027 32074 tablet_bootstrap.cc:654] T 7343c21e7dfe47a2bfcc793de456d971 P e9c07145e2d845b680ba06c14c6e0e5c: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:26.535797 32075 tablet_bootstrap.cc:492] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6: No bootstrap required, opened a new log
I20240502 07:58:26.535900 32075 ts_tablet_manager.cc:1367] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 07:58:26.536170 32074 tablet_bootstrap.cc:492] T 7343c21e7dfe47a2bfcc793de456d971 P e9c07145e2d845b680ba06c14c6e0e5c: No bootstrap required, opened a new log
I20240502 07:58:26.536286 32074 ts_tablet_manager.cc:1367] T 7343c21e7dfe47a2bfcc793de456d971 P e9c07145e2d845b680ba06c14c6e0e5c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:26.536383 32075 raft_consensus.cc:348] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } } peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } }
I20240502 07:58:26.536514 32075 raft_consensus.cc:374] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:26.536559 32075 raft_consensus.cc:729] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7feef95690fe4b0bb736a884af9569e6, State: Initialized, Role: FOLLOWER
I20240502 07:58:26.536703 32075 consensus_queue.cc:260] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } } peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } }
I20240502 07:58:26.536770 32074 raft_consensus.cc:348] T 7343c21e7dfe47a2bfcc793de456d971 P e9c07145e2d845b680ba06c14c6e0e5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } } peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } }
I20240502 07:58:26.536875 32074 raft_consensus.cc:374] T 7343c21e7dfe47a2bfcc793de456d971 P e9c07145e2d845b680ba06c14c6e0e5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:26.536908 32074 raft_consensus.cc:729] T 7343c21e7dfe47a2bfcc793de456d971 P e9c07145e2d845b680ba06c14c6e0e5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c07145e2d845b680ba06c14c6e0e5c, State: Initialized, Role: FOLLOWER
I20240502 07:58:26.536962 32075 ts_tablet_manager.cc:1398] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:26.537034 32074 consensus_queue.cc:260] T 7343c21e7dfe47a2bfcc793de456d971 P e9c07145e2d845b680ba06c14c6e0e5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } } peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } }
I20240502 07:58:26.537299 32074 ts_tablet_manager.cc:1398] T 7343c21e7dfe47a2bfcc793de456d971 P e9c07145e2d845b680ba06c14c6e0e5c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:26.537572 32076 raft_consensus.cc:348] T 862e4df4c8004aa0973a06875b587e5f P f9e57376395e46fb8e35ed8f1b613695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } } peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } }
I20240502 07:58:26.537679 32076 raft_consensus.cc:374] T 862e4df4c8004aa0973a06875b587e5f P f9e57376395e46fb8e35ed8f1b613695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:26.537714 32076 raft_consensus.cc:729] T 862e4df4c8004aa0973a06875b587e5f P f9e57376395e46fb8e35ed8f1b613695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e57376395e46fb8e35ed8f1b613695, State: Initialized, Role: FOLLOWER
I20240502 07:58:26.537868 32076 consensus_queue.cc:260] T 862e4df4c8004aa0973a06875b587e5f P f9e57376395e46fb8e35ed8f1b613695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } } peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } }
I20240502 07:58:26.538295 32076 ts_tablet_manager.cc:1398] T 862e4df4c8004aa0973a06875b587e5f P f9e57376395e46fb8e35ed8f1b613695: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 07:58:26.538583 32076 tablet_bootstrap.cc:492] T 7343c21e7dfe47a2bfcc793de456d971 P f9e57376395e46fb8e35ed8f1b613695: Bootstrap starting.
I20240502 07:58:26.539798 32076 tablet_bootstrap.cc:654] T 7343c21e7dfe47a2bfcc793de456d971 P f9e57376395e46fb8e35ed8f1b613695: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:26.541366 32076 tablet_bootstrap.cc:492] T 7343c21e7dfe47a2bfcc793de456d971 P f9e57376395e46fb8e35ed8f1b613695: No bootstrap required, opened a new log
I20240502 07:58:26.541456 32076 ts_tablet_manager.cc:1367] T 7343c21e7dfe47a2bfcc793de456d971 P f9e57376395e46fb8e35ed8f1b613695: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:26.541877 32076 raft_consensus.cc:348] T 7343c21e7dfe47a2bfcc793de456d971 P f9e57376395e46fb8e35ed8f1b613695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } } peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } }
I20240502 07:58:26.541967 32076 raft_consensus.cc:374] T 7343c21e7dfe47a2bfcc793de456d971 P f9e57376395e46fb8e35ed8f1b613695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:26.542003 32076 raft_consensus.cc:729] T 7343c21e7dfe47a2bfcc793de456d971 P f9e57376395e46fb8e35ed8f1b613695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e57376395e46fb8e35ed8f1b613695, State: Initialized, Role: FOLLOWER
I20240502 07:58:26.542135 32076 consensus_queue.cc:260] T 7343c21e7dfe47a2bfcc793de456d971 P f9e57376395e46fb8e35ed8f1b613695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } } peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } }
I20240502 07:58:26.542366 32076 ts_tablet_manager.cc:1398] T 7343c21e7dfe47a2bfcc793de456d971 P f9e57376395e46fb8e35ed8f1b613695: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:26.631721 32078 raft_consensus.cc:482] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:26.631922 32078 raft_consensus.cc:504] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } } peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } }
I20240502 07:58:26.632577 32078 leader_election.cc:288] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9c07145e2d845b680ba06c14c6e0e5c (127.29.106.129:42333), f9e57376395e46fb8e35ed8f1b613695 (127.29.106.130:39621)
I20240502 07:58:26.638121 31877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7343c21e7dfe47a2bfcc793de456d971" candidate_uuid: "7feef95690fe4b0bb736a884af9569e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" is_pre_election: true
I20240502 07:58:26.638116 31953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7343c21e7dfe47a2bfcc793de456d971" candidate_uuid: "7feef95690fe4b0bb736a884af9569e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e57376395e46fb8e35ed8f1b613695" is_pre_election: true
I20240502 07:58:26.638368 31877 raft_consensus.cc:2454] T 7343c21e7dfe47a2bfcc793de456d971 P e9c07145e2d845b680ba06c14c6e0e5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7feef95690fe4b0bb736a884af9569e6 in term 0.
I20240502 07:58:26.638368 31953 raft_consensus.cc:2454] T 7343c21e7dfe47a2bfcc793de456d971 P f9e57376395e46fb8e35ed8f1b613695 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7feef95690fe4b0bb736a884af9569e6 in term 0.
I20240502 07:58:26.638792 31993 leader_election.cc:302] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7feef95690fe4b0bb736a884af9569e6, f9e57376395e46fb8e35ed8f1b613695; no voters:
I20240502 07:58:26.639057 32078 raft_consensus.cc:2781] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:26.639186 32078 raft_consensus.cc:482] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:26.639243 32078 raft_consensus.cc:3037] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:26.641496 32088 raft_consensus.cc:482] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:26.641626 32088 raft_consensus.cc:504] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } } peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } }
I20240502 07:58:26.642114 32088 leader_election.cc:288] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9c07145e2d845b680ba06c14c6e0e5c (127.29.106.129:42333), f9e57376395e46fb8e35ed8f1b613695 (127.29.106.130:39621)
I20240502 07:58:26.642617 31953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "862e4df4c8004aa0973a06875b587e5f" candidate_uuid: "7feef95690fe4b0bb736a884af9569e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e57376395e46fb8e35ed8f1b613695" is_pre_election: true
I20240502 07:58:26.642617 31877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "862e4df4c8004aa0973a06875b587e5f" candidate_uuid: "7feef95690fe4b0bb736a884af9569e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" is_pre_election: true
I20240502 07:58:26.642833 31953 raft_consensus.cc:2454] T 862e4df4c8004aa0973a06875b587e5f P f9e57376395e46fb8e35ed8f1b613695 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7feef95690fe4b0bb736a884af9569e6 in term 0.
I20240502 07:58:26.642833 31877 raft_consensus.cc:2454] T 862e4df4c8004aa0973a06875b587e5f P e9c07145e2d845b680ba06c14c6e0e5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7feef95690fe4b0bb736a884af9569e6 in term 0.
I20240502 07:58:26.643253 31992 leader_election.cc:302] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7feef95690fe4b0bb736a884af9569e6, e9c07145e2d845b680ba06c14c6e0e5c, f9e57376395e46fb8e35ed8f1b613695; no voters:
I20240502 07:58:26.643589 32088 raft_consensus.cc:2781] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:26.643647 32088 raft_consensus.cc:482] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:26.643676 32088 raft_consensus.cc:3037] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:26.652228 32078 raft_consensus.cc:504] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } } peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } }
I20240502 07:58:26.652909 32078 leader_election.cc:288] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6 [CANDIDATE]: Term 1 election: Requested vote from peers e9c07145e2d845b680ba06c14c6e0e5c (127.29.106.129:42333), f9e57376395e46fb8e35ed8f1b613695 (127.29.106.130:39621)
I20240502 07:58:26.653465 31877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7343c21e7dfe47a2bfcc793de456d971" candidate_uuid: "7feef95690fe4b0bb736a884af9569e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c07145e2d845b680ba06c14c6e0e5c"
I20240502 07:58:26.653465 31953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7343c21e7dfe47a2bfcc793de456d971" candidate_uuid: "7feef95690fe4b0bb736a884af9569e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e57376395e46fb8e35ed8f1b613695"
I20240502 07:58:26.653712 31877 raft_consensus.cc:3037] T 7343c21e7dfe47a2bfcc793de456d971 P e9c07145e2d845b680ba06c14c6e0e5c [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:26.653728 31953 raft_consensus.cc:3037] T 7343c21e7dfe47a2bfcc793de456d971 P f9e57376395e46fb8e35ed8f1b613695 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:26.668637 32088 raft_consensus.cc:504] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } } peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } }
I20240502 07:58:26.668711 31953 raft_consensus.cc:2454] T 7343c21e7dfe47a2bfcc793de456d971 P f9e57376395e46fb8e35ed8f1b613695 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7feef95690fe4b0bb736a884af9569e6 in term 1.
I20240502 07:58:26.668711 31877 raft_consensus.cc:2454] T 7343c21e7dfe47a2bfcc793de456d971 P e9c07145e2d845b680ba06c14c6e0e5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7feef95690fe4b0bb736a884af9569e6 in term 1.
I20240502 07:58:26.669320 32088 leader_election.cc:288] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [CANDIDATE]: Term 1 election: Requested vote from peers e9c07145e2d845b680ba06c14c6e0e5c (127.29.106.129:42333), f9e57376395e46fb8e35ed8f1b613695 (127.29.106.130:39621)
I20240502 07:58:26.669545 31992 leader_election.cc:302] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7feef95690fe4b0bb736a884af9569e6, e9c07145e2d845b680ba06c14c6e0e5c; no voters:
I20240502 07:58:26.669799 32088 raft_consensus.cc:2781] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:26.669770 31877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "862e4df4c8004aa0973a06875b587e5f" candidate_uuid: "7feef95690fe4b0bb736a884af9569e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c07145e2d845b680ba06c14c6e0e5c"
I20240502 07:58:26.669955 31877 raft_consensus.cc:3037] T 862e4df4c8004aa0973a06875b587e5f P e9c07145e2d845b680ba06c14c6e0e5c [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:26.669929 32088 raft_consensus.cc:686] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6 [term 1 LEADER]: Becoming Leader. State: Replica: 7feef95690fe4b0bb736a884af9569e6, State: Running, Role: LEADER
I20240502 07:58:26.669934 31953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "862e4df4c8004aa0973a06875b587e5f" candidate_uuid: "7feef95690fe4b0bb736a884af9569e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e57376395e46fb8e35ed8f1b613695"
I20240502 07:58:26.670137 31953 raft_consensus.cc:3037] T 862e4df4c8004aa0973a06875b587e5f P f9e57376395e46fb8e35ed8f1b613695 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:26.670157 32088 consensus_queue.cc:237] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } } peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } }
I20240502 07:58:26.672039 31774 catalog_manager.cc:5484] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7feef95690fe4b0bb736a884af9569e6 (127.29.106.131). New cstate: current_term: 1 leader_uuid: "7feef95690fe4b0bb736a884af9569e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:26.682600 31877 raft_consensus.cc:2454] T 862e4df4c8004aa0973a06875b587e5f P e9c07145e2d845b680ba06c14c6e0e5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7feef95690fe4b0bb736a884af9569e6 in term 1.
I20240502 07:58:26.682615 31953 raft_consensus.cc:2454] T 862e4df4c8004aa0973a06875b587e5f P f9e57376395e46fb8e35ed8f1b613695 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7feef95690fe4b0bb736a884af9569e6 in term 1.
I20240502 07:58:26.683092 31992 leader_election.cc:302] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7feef95690fe4b0bb736a884af9569e6, e9c07145e2d845b680ba06c14c6e0e5c; no voters:
I20240502 07:58:26.683329 32088 raft_consensus.cc:2781] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:26.683425 32088 raft_consensus.cc:686] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [term 1 LEADER]: Becoming Leader. State: Replica: 7feef95690fe4b0bb736a884af9569e6, State: Running, Role: LEADER
I20240502 07:58:26.683583 32088 consensus_queue.cc:237] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } } peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } }
I20240502 07:58:26.685623 31774 catalog_manager.cc:5484] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7feef95690fe4b0bb736a884af9569e6 (127.29.106.131). New cstate: current_term: 1 leader_uuid: "7feef95690fe4b0bb736a884af9569e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:27.073665 32088 consensus_queue.cc:1035] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:27.077220 32078 consensus_queue.cc:1035] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:27.182430 32078 consensus_queue.cc:1035] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:27.185595 32078 consensus_queue.cc:1035] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:27.352943 31828 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:58:27.358392 32028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7343c21e7dfe47a2bfcc793de456d971"
dest_uuid: "7feef95690fe4b0bb736a884af9569e6"
mode: GRACEFUL
new_leader_uuid: "e9c07145e2d845b680ba06c14c6e0e5c"
from {username='slave'} at 127.0.0.1:33896
I20240502 07:58:27.358645 32028 raft_consensus.cc:595] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6 [term 1 LEADER]: Received request to transfer leadership to TS e9c07145e2d845b680ba06c14c6e0e5c
I20240502 07:58:27.359177 31828 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240502 07:58:27.359335 31828 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:58:27.511998 32088 raft_consensus.cc:979] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6: : Instructing follower e9c07145e2d845b680ba06c14c6e0e5c to start an election
I20240502 07:58:27.512185 32088 raft_consensus.cc:1067] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6 [term 1 LEADER]: Signalling peer e9c07145e2d845b680ba06c14c6e0e5c to start an election
I20240502 07:58:27.512761 31877 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7343c21e7dfe47a2bfcc793de456d971"
dest_uuid: "e9c07145e2d845b680ba06c14c6e0e5c"
from {username='slave'} at 127.0.0.1:52108
I20240502 07:58:27.512921 31877 raft_consensus.cc:482] T 7343c21e7dfe47a2bfcc793de456d971 P e9c07145e2d845b680ba06c14c6e0e5c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:58:27.512996 31877 raft_consensus.cc:3037] T 7343c21e7dfe47a2bfcc793de456d971 P e9c07145e2d845b680ba06c14c6e0e5c [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:27.514035 31877 raft_consensus.cc:504] T 7343c21e7dfe47a2bfcc793de456d971 P e9c07145e2d845b680ba06c14c6e0e5c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } } peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } }
I20240502 07:58:27.514544 31877 leader_election.cc:288] T 7343c21e7dfe47a2bfcc793de456d971 P e9c07145e2d845b680ba06c14c6e0e5c [CANDIDATE]: Term 2 election: Requested vote from peers f9e57376395e46fb8e35ed8f1b613695 (127.29.106.130:39621), 7feef95690fe4b0bb736a884af9569e6 (127.29.106.131:41637)
I20240502 07:58:27.519703 31953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7343c21e7dfe47a2bfcc793de456d971" candidate_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f9e57376395e46fb8e35ed8f1b613695"
I20240502 07:58:27.519894 31953 raft_consensus.cc:3037] T 7343c21e7dfe47a2bfcc793de456d971 P f9e57376395e46fb8e35ed8f1b613695 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:27.520407 32028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7343c21e7dfe47a2bfcc793de456d971" candidate_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7feef95690fe4b0bb736a884af9569e6"
I20240502 07:58:27.520570 32028 raft_consensus.cc:3032] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:58:27.520624 32028 raft_consensus.cc:729] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7feef95690fe4b0bb736a884af9569e6, State: Running, Role: LEADER
I20240502 07:58:27.520857 32028 consensus_queue.cc:260] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } } peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } }
I20240502 07:58:27.520983 31953 raft_consensus.cc:2454] T 7343c21e7dfe47a2bfcc793de456d971 P f9e57376395e46fb8e35ed8f1b613695 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c07145e2d845b680ba06c14c6e0e5c in term 2.
I20240502 07:58:27.521060 32028 raft_consensus.cc:3037] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:27.521359 31842 leader_election.cc:302] T 7343c21e7dfe47a2bfcc793de456d971 P e9c07145e2d845b680ba06c14c6e0e5c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e9c07145e2d845b680ba06c14c6e0e5c, f9e57376395e46fb8e35ed8f1b613695; no voters:
I20240502 07:58:27.521576 32095 raft_consensus.cc:2781] T 7343c21e7dfe47a2bfcc793de456d971 P e9c07145e2d845b680ba06c14c6e0e5c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:58:27.521786 32095 raft_consensus.cc:686] T 7343c21e7dfe47a2bfcc793de456d971 P e9c07145e2d845b680ba06c14c6e0e5c [term 2 LEADER]: Becoming Leader. State: Replica: e9c07145e2d845b680ba06c14c6e0e5c, State: Running, Role: LEADER
I20240502 07:58:27.521979 32095 consensus_queue.cc:237] T 7343c21e7dfe47a2bfcc793de456d971 P e9c07145e2d845b680ba06c14c6e0e5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } } peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } }
I20240502 07:58:27.522728 32028 raft_consensus.cc:2454] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c07145e2d845b680ba06c14c6e0e5c in term 2.
I20240502 07:58:27.523521 31774 catalog_manager.cc:5484] T 7343c21e7dfe47a2bfcc793de456d971 P e9c07145e2d845b680ba06c14c6e0e5c reported cstate change: term changed from 1 to 2, leader changed from 7feef95690fe4b0bb736a884af9569e6 (127.29.106.131) to e9c07145e2d845b680ba06c14c6e0e5c (127.29.106.129). New cstate: current_term: 2 leader_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:27.942828 31953 raft_consensus.cc:1261] T 7343c21e7dfe47a2bfcc793de456d971 P f9e57376395e46fb8e35ed8f1b613695 [term 2 FOLLOWER]: Refusing update from remote peer e9c07145e2d845b680ba06c14c6e0e5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 07:58:27.943679 32095 consensus_queue.cc:1035] T 7343c21e7dfe47a2bfcc793de456d971 P e9c07145e2d845b680ba06c14c6e0e5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 07:58:27.946568 32028 raft_consensus.cc:1261] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6 [term 2 FOLLOWER]: Refusing update from remote peer e9c07145e2d845b680ba06c14c6e0e5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 07:58:27.947158 32106 consensus_queue.cc:1035] T 7343c21e7dfe47a2bfcc793de456d971 P e9c07145e2d845b680ba06c14c6e0e5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 07:58:29.359733 31828 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:58:29.360416 31828 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 07:58:29.360498 31828 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:58:31.360874 31828 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:58:31.361285 31828 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 07:58:31.361380 31828 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:58:33.361631 31828 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:58:33.362010 31828 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 07:58:33.362078 31828 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:58:35.362478 31828 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:58:35.362990 31828 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 07:58:35.363077 31828 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:58:37.363435 31828 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:58:37.363826 31828 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 07:58:37.363893 31828 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:58:37.767779 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:37.771309 32120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:37.772051 32117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:37.772694 30122 server_base.cc:1034] running on GCE node
W20240502 07:58:37.773608 32118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:37.773917 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:58:37.773972 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:58:37.773988 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636717773988 us; error 0 us; skew 500 ppm
I20240502 07:58:37.775341 30122 webserver.cc:460] Webserver started at http://127.29.106.132:41643/ using document root <none> and password file <none>
I20240502 07:58:37.775523 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:37.775589 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:37.775682 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:37.776118 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636688849691-30122-0/minicluster-data/ts-3-root/instance:
uuid: "e11ea6ca87794bbc8858fe02903eb3bb"
format_stamp: "Formatted at 2024-05-02 07:58:37 on dist-test-slave-m6l2"
I20240502 07:58:37.777631 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:37.778559 32125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:37.778813 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:37.778883 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636688849691-30122-0/minicluster-data/ts-3-root
uuid: "e11ea6ca87794bbc8858fe02903eb3bb"
format_stamp: "Formatted at 2024-05-02 07:58:37 on dist-test-slave-m6l2"
I20240502 07:58:37.778947 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636688849691-30122-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636688849691-30122-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636688849691-30122-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:37.799120 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:37.799633 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:37.800005 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:37.800621 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:37.800668 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:37.800721 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:37.800752 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:37.806394 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.132:36823
I20240502 07:58:37.806489 32188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.132:36823 every 8 connection(s)
I20240502 07:58:37.811971 32189 heartbeater.cc:347] Connected to a master server at 127.29.106.190:40157
I20240502 07:58:37.812106 32189 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:37.812389 32189 heartbeater.cc:511] Master 127.29.106.190:40157 requested a full tablet report, sending...
I20240502 07:58:37.813072 31774 ts_manager.cc:194] Registered new tserver with Master: e11ea6ca87794bbc8858fe02903eb3bb (127.29.106.132:36823)
I20240502 07:58:37.813735 31774 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39224
I20240502 07:58:38.376016 32028 consensus_queue.cc:237] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } } peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "e11ea6ca87794bbc8858fe02903eb3bb" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 36823 } attrs { promote: true } }
I20240502 07:58:38.377873 31953 raft_consensus.cc:1261] T 862e4df4c8004aa0973a06875b587e5f P f9e57376395e46fb8e35ed8f1b613695 [term 1 FOLLOWER]: Refusing update from remote peer 7feef95690fe4b0bb736a884af9569e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:38.377873 31877 raft_consensus.cc:1261] T 862e4df4c8004aa0973a06875b587e5f P e9c07145e2d845b680ba06c14c6e0e5c [term 1 FOLLOWER]: Refusing update from remote peer 7feef95690fe4b0bb736a884af9569e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:38.378509 32197 consensus_queue.cc:1035] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 07:58:38.378871 32196 consensus_queue.cc:1035] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 07:58:38.380710 32197 raft_consensus.cc:2932] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e11ea6ca87794bbc8858fe02903eb3bb (127.29.106.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } } peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "e11ea6ca87794bbc8858fe02903eb3bb" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 36823 } attrs { promote: true } } }
I20240502 07:58:38.382175 31877 raft_consensus.cc:2932] T 862e4df4c8004aa0973a06875b587e5f P e9c07145e2d845b680ba06c14c6e0e5c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e11ea6ca87794bbc8858fe02903eb3bb (127.29.106.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } } peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "e11ea6ca87794bbc8858fe02903eb3bb" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 36823 } attrs { promote: true } } }
W20240502 07:58:38.383005 31994 consensus_peers.cc:485] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 -> Peer e11ea6ca87794bbc8858fe02903eb3bb (127.29.106.132:36823): Couldn't send request to peer e11ea6ca87794bbc8858fe02903eb3bb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 862e4df4c8004aa0973a06875b587e5f. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:38.384028 31953 raft_consensus.cc:2932] T 862e4df4c8004aa0973a06875b587e5f P f9e57376395e46fb8e35ed8f1b613695 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e11ea6ca87794bbc8858fe02903eb3bb (127.29.106.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } } peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "e11ea6ca87794bbc8858fe02903eb3bb" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 36823 } attrs { promote: true } } }
I20240502 07:58:38.384691 31772 catalog_manager.cc:5484] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 reported cstate change: config changed from index -1 to 2, NON_VOTER e11ea6ca87794bbc8858fe02903eb3bb (127.29.106.132) added. New cstate: current_term: 1 leader_uuid: "7feef95690fe4b0bb736a884af9569e6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e11ea6ca87794bbc8858fe02903eb3bb" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 36823 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:38.814705 32189 heartbeater.cc:503] Master 127.29.106.190:40157 was elected leader, sending a full tablet report...
I20240502 07:58:38.818365 32208 ts_tablet_manager.cc:902] T 862e4df4c8004aa0973a06875b587e5f P e11ea6ca87794bbc8858fe02903eb3bb: Initiating tablet copy from peer 7feef95690fe4b0bb736a884af9569e6 (127.29.106.131:41637)
I20240502 07:58:38.819265 32208 tablet_copy_client.cc:308] T 862e4df4c8004aa0973a06875b587e5f P e11ea6ca87794bbc8858fe02903eb3bb: tablet copy: Beginning tablet copy session from remote peer at address 127.29.106.131:41637
I20240502 07:58:38.822925 32038 tablet_copy_service.cc:140] P 7feef95690fe4b0bb736a884af9569e6: Received BeginTabletCopySession request for tablet 862e4df4c8004aa0973a06875b587e5f from peer e11ea6ca87794bbc8858fe02903eb3bb ({username='slave'} at 127.0.0.1:48634)
I20240502 07:58:38.823071 32038 tablet_copy_service.cc:161] P 7feef95690fe4b0bb736a884af9569e6: Beginning new tablet copy session on tablet 862e4df4c8004aa0973a06875b587e5f from peer e11ea6ca87794bbc8858fe02903eb3bb at {username='slave'} at 127.0.0.1:48634: session id = e11ea6ca87794bbc8858fe02903eb3bb-862e4df4c8004aa0973a06875b587e5f
I20240502 07:58:38.824249 32038 tablet_copy_source_session.cc:205] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:38.825531 32208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 862e4df4c8004aa0973a06875b587e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:38.828084 32208 tablet_copy_client.cc:783] T 862e4df4c8004aa0973a06875b587e5f P e11ea6ca87794bbc8858fe02903eb3bb: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:38.828395 32208 tablet_copy_client.cc:647] T 862e4df4c8004aa0973a06875b587e5f P e11ea6ca87794bbc8858fe02903eb3bb: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:38.830106 32208 tablet_copy_client.cc:523] T 862e4df4c8004aa0973a06875b587e5f P e11ea6ca87794bbc8858fe02903eb3bb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:38.833323 32208 tablet_bootstrap.cc:492] T 862e4df4c8004aa0973a06875b587e5f P e11ea6ca87794bbc8858fe02903eb3bb: Bootstrap starting.
I20240502 07:58:38.838908 32208 tablet_bootstrap.cc:492] T 862e4df4c8004aa0973a06875b587e5f P e11ea6ca87794bbc8858fe02903eb3bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:38.839293 32208 tablet_bootstrap.cc:492] T 862e4df4c8004aa0973a06875b587e5f P e11ea6ca87794bbc8858fe02903eb3bb: Bootstrap complete.
I20240502 07:58:38.839495 32208 ts_tablet_manager.cc:1367] T 862e4df4c8004aa0973a06875b587e5f P e11ea6ca87794bbc8858fe02903eb3bb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20240502 07:58:38.840070 32208 raft_consensus.cc:348] T 862e4df4c8004aa0973a06875b587e5f P e11ea6ca87794bbc8858fe02903eb3bb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } } peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "e11ea6ca87794bbc8858fe02903eb3bb" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 36823 } attrs { promote: true } }
I20240502 07:58:38.840240 32208 raft_consensus.cc:729] T 862e4df4c8004aa0973a06875b587e5f P e11ea6ca87794bbc8858fe02903eb3bb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e11ea6ca87794bbc8858fe02903eb3bb, State: Initialized, Role: LEARNER
I20240502 07:58:38.840396 32208 consensus_queue.cc:260] T 862e4df4c8004aa0973a06875b587e5f P e11ea6ca87794bbc8858fe02903eb3bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } } peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "e11ea6ca87794bbc8858fe02903eb3bb" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 36823 } attrs { promote: true } }
I20240502 07:58:38.840862 32208 ts_tablet_manager.cc:1398] T 862e4df4c8004aa0973a06875b587e5f P e11ea6ca87794bbc8858fe02903eb3bb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:38.841441 32038 tablet_copy_service.cc:342] P 7feef95690fe4b0bb736a884af9569e6: Request end of tablet copy session e11ea6ca87794bbc8858fe02903eb3bb-862e4df4c8004aa0973a06875b587e5f received from {username='slave'} at 127.0.0.1:48634
I20240502 07:58:38.841535 32038 tablet_copy_service.cc:433] P 7feef95690fe4b0bb736a884af9569e6: ending tablet copy session e11ea6ca87794bbc8858fe02903eb3bb-862e4df4c8004aa0973a06875b587e5f on tablet 862e4df4c8004aa0973a06875b587e5f with peer e11ea6ca87794bbc8858fe02903eb3bb
I20240502 07:58:39.364122 31828 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:58:39.364543 31828 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 07:58:39.364611 31828 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:58:39.418749 32163 raft_consensus.cc:1203] T 862e4df4c8004aa0973a06875b587e5f P e11ea6ca87794bbc8858fe02903eb3bb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 07:58:40.002563 32217 raft_consensus.cc:1050] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6: attempting to promote NON_VOTER e11ea6ca87794bbc8858fe02903eb3bb to VOTER
I20240502 07:58:40.003118 32217 consensus_queue.cc:237] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } } peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "e11ea6ca87794bbc8858fe02903eb3bb" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 36823 } attrs { promote: false } }
I20240502 07:58:40.004410 32163 raft_consensus.cc:1261] T 862e4df4c8004aa0973a06875b587e5f P e11ea6ca87794bbc8858fe02903eb3bb [term 1 LEARNER]: Refusing update from remote peer 7feef95690fe4b0bb736a884af9569e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 07:58:40.004801 31953 raft_consensus.cc:1261] T 862e4df4c8004aa0973a06875b587e5f P f9e57376395e46fb8e35ed8f1b613695 [term 1 FOLLOWER]: Refusing update from remote peer 7feef95690fe4b0bb736a884af9569e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 07:58:40.005065 31877 raft_consensus.cc:1261] T 862e4df4c8004aa0973a06875b587e5f P e9c07145e2d845b680ba06c14c6e0e5c [term 1 FOLLOWER]: Refusing update from remote peer 7feef95690fe4b0bb736a884af9569e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 07:58:40.005084 32221 consensus_queue.cc:1035] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e11ea6ca87794bbc8858fe02903eb3bb" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 36823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 07:58:40.005440 32217 consensus_queue.cc:1035] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 07:58:40.005612 32219 consensus_queue.cc:1035] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 07:58:40.007670 32217 raft_consensus.cc:2932] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e11ea6ca87794bbc8858fe02903eb3bb (127.29.106.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } } peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "e11ea6ca87794bbc8858fe02903eb3bb" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 36823 } attrs { promote: false } } }
I20240502 07:58:40.008210 31953 raft_consensus.cc:2932] T 862e4df4c8004aa0973a06875b587e5f P f9e57376395e46fb8e35ed8f1b613695 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e11ea6ca87794bbc8858fe02903eb3bb (127.29.106.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } } peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "e11ea6ca87794bbc8858fe02903eb3bb" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 36823 } attrs { promote: false } } }
I20240502 07:58:40.008231 31877 raft_consensus.cc:2932] T 862e4df4c8004aa0973a06875b587e5f P e9c07145e2d845b680ba06c14c6e0e5c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e11ea6ca87794bbc8858fe02903eb3bb (127.29.106.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } } peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "e11ea6ca87794bbc8858fe02903eb3bb" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 36823 } attrs { promote: false } } }
I20240502 07:58:40.010077 32163 raft_consensus.cc:2932] T 862e4df4c8004aa0973a06875b587e5f P e11ea6ca87794bbc8858fe02903eb3bb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e11ea6ca87794bbc8858fe02903eb3bb (127.29.106.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } } peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "e11ea6ca87794bbc8858fe02903eb3bb" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 36823 } attrs { promote: false } } }
I20240502 07:58:40.010233 31773 catalog_manager.cc:5484] T 862e4df4c8004aa0973a06875b587e5f P e9c07145e2d845b680ba06c14c6e0e5c reported cstate change: config changed from index 2 to 3, e11ea6ca87794bbc8858fe02903eb3bb (127.29.106.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7feef95690fe4b0bb736a884af9569e6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } } peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "f9e57376395e46fb8e35ed8f1b613695" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "e11ea6ca87794bbc8858fe02903eb3bb" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 36823 } attrs { promote: false } } }
I20240502 07:58:40.015198 32028 consensus_queue.cc:237] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } } peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "e11ea6ca87794bbc8858fe02903eb3bb" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 36823 } attrs { promote: false } }
I20240502 07:58:40.016330 32163 raft_consensus.cc:1261] T 862e4df4c8004aa0973a06875b587e5f P e11ea6ca87794bbc8858fe02903eb3bb [term 1 FOLLOWER]: Refusing update from remote peer 7feef95690fe4b0bb736a884af9569e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:58:40.016366 31877 raft_consensus.cc:1261] T 862e4df4c8004aa0973a06875b587e5f P e9c07145e2d845b680ba06c14c6e0e5c [term 1 FOLLOWER]: Refusing update from remote peer 7feef95690fe4b0bb736a884af9569e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:58:40.016707 32219 consensus_queue.cc:1035] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:58:40.016891 32217 consensus_queue.cc:1035] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e11ea6ca87794bbc8858fe02903eb3bb" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 36823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:58:40.018201 32217 raft_consensus.cc:2932] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f9e57376395e46fb8e35ed8f1b613695 (127.29.106.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } } peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "e11ea6ca87794bbc8858fe02903eb3bb" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 36823 } attrs { promote: false } } }
I20240502 07:58:40.018591 31877 raft_consensus.cc:2932] T 862e4df4c8004aa0973a06875b587e5f P e9c07145e2d845b680ba06c14c6e0e5c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f9e57376395e46fb8e35ed8f1b613695 (127.29.106.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } } peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "e11ea6ca87794bbc8858fe02903eb3bb" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 36823 } attrs { promote: false } } }
I20240502 07:58:40.018707 32163 raft_consensus.cc:2932] T 862e4df4c8004aa0973a06875b587e5f P e11ea6ca87794bbc8858fe02903eb3bb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f9e57376395e46fb8e35ed8f1b613695 (127.29.106.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } } peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "e11ea6ca87794bbc8858fe02903eb3bb" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 36823 } attrs { promote: false } } }
I20240502 07:58:40.019732 31760 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 862e4df4c8004aa0973a06875b587e5f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:40.021116 31771 catalog_manager.cc:5484] T 862e4df4c8004aa0973a06875b587e5f P e11ea6ca87794bbc8858fe02903eb3bb reported cstate change: config changed from index 3 to 4, VOTER f9e57376395e46fb8e35ed8f1b613695 (127.29.106.130) evicted. New cstate: current_term: 1 leader_uuid: "7feef95690fe4b0bb736a884af9569e6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7feef95690fe4b0bb736a884af9569e6" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 41637 } } peers { permanent_uuid: "e9c07145e2d845b680ba06c14c6e0e5c" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 42333 } } peers { permanent_uuid: "e11ea6ca87794bbc8858fe02903eb3bb" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 36823 } attrs { promote: false } } }
I20240502 07:58:40.023792 31943 tablet_service.cc:1514] Processing DeleteTablet for tablet 862e4df4c8004aa0973a06875b587e5f with delete_type TABLET_DATA_TOMBSTONED (TS f9e57376395e46fb8e35ed8f1b613695 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60722
I20240502 07:58:40.024322 32232 tablet_replica.cc:330] T 862e4df4c8004aa0973a06875b587e5f P f9e57376395e46fb8e35ed8f1b613695: stopping tablet replica
I20240502 07:58:40.024451 32232 raft_consensus.cc:2229] T 862e4df4c8004aa0973a06875b587e5f P f9e57376395e46fb8e35ed8f1b613695 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:40.024605 32232 raft_consensus.cc:2258] T 862e4df4c8004aa0973a06875b587e5f P f9e57376395e46fb8e35ed8f1b613695 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:40.025141 32232 ts_tablet_manager.cc:1875] T 862e4df4c8004aa0973a06875b587e5f P f9e57376395e46fb8e35ed8f1b613695: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:40.027027 32232 ts_tablet_manager.cc:1888] T 862e4df4c8004aa0973a06875b587e5f P f9e57376395e46fb8e35ed8f1b613695: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 07:58:40.027114 32232 log.cc:1196] T 862e4df4c8004aa0973a06875b587e5f P f9e57376395e46fb8e35ed8f1b613695: Deleting WAL directory at /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636688849691-30122-0/minicluster-data/ts-1-root/wals/862e4df4c8004aa0973a06875b587e5f
I20240502 07:58:40.027752 31760 catalog_manager.cc:4830] TS f9e57376395e46fb8e35ed8f1b613695 (127.29.106.130:39621): tablet 862e4df4c8004aa0973a06875b587e5f (table test-workload [id=9927fd31327440198002a2789860edcc]) successfully deleted
I20240502 07:58:41.365005 31828 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:58:41.365578 31828 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 07:58:41.365658 31828 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:58:43.366144 31828 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:58:43.366693 31828 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 07:58:43.366783 31828 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:58:45.367188 31828 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:58:45.367852 31828 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 07:58:45.367945 31828 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:58:47.368520 31828 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:58:47.369062 31828 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 07:58:47.369146 31828 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:58:49.369518 31828 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:58:49.369995 31828 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 07:58:49.370069 31828 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:58:51.370548 31828 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:58:51.371069 31828 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 07:58:51.371143 31828 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:58:51.884910 30122 tablet_server.cc:178] TabletServer@127.29.106.129:0 shutting down...
I20240502 07:58:51.890008 30122 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 07:58:51.890300 30122 tablet_replica.cc:330] T 7343c21e7dfe47a2bfcc793de456d971 P e9c07145e2d845b680ba06c14c6e0e5c: stopping tablet replica
I20240502 07:58:51.890455 30122 raft_consensus.cc:2229] T 7343c21e7dfe47a2bfcc793de456d971 P e9c07145e2d845b680ba06c14c6e0e5c [term 2 LEADER]: Raft consensus shutting down.
I20240502 07:58:51.890750 30122 raft_consensus.cc:2258] T 7343c21e7dfe47a2bfcc793de456d971 P e9c07145e2d845b680ba06c14c6e0e5c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:51.891331 30122 tablet_replica.cc:330] T 862e4df4c8004aa0973a06875b587e5f P e9c07145e2d845b680ba06c14c6e0e5c: stopping tablet replica
I20240502 07:58:51.891443 30122 raft_consensus.cc:2229] T 862e4df4c8004aa0973a06875b587e5f P e9c07145e2d845b680ba06c14c6e0e5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:51.891574 30122 raft_consensus.cc:2258] T 862e4df4c8004aa0973a06875b587e5f P e9c07145e2d845b680ba06c14c6e0e5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:51.905247 30122 tablet_server.cc:194] TabletServer@127.29.106.129:0 shutdown complete.
I20240502 07:58:51.908012 30122 tablet_server.cc:178] TabletServer@127.29.106.130:0 shutting down...
I20240502 07:58:51.913578 30122 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 07:58:51.913820 30122 tablet_replica.cc:330] T 7343c21e7dfe47a2bfcc793de456d971 P f9e57376395e46fb8e35ed8f1b613695: stopping tablet replica
I20240502 07:58:51.913962 30122 raft_consensus.cc:2229] T 7343c21e7dfe47a2bfcc793de456d971 P f9e57376395e46fb8e35ed8f1b613695 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:51.914105 30122 raft_consensus.cc:2258] T 7343c21e7dfe47a2bfcc793de456d971 P f9e57376395e46fb8e35ed8f1b613695 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:51.928953 30122 tablet_server.cc:194] TabletServer@127.29.106.130:0 shutdown complete.
I20240502 07:58:51.931998 30122 tablet_server.cc:178] TabletServer@127.29.106.131:0 shutting down...
I20240502 07:58:51.937911 30122 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 07:58:51.938187 30122 tablet_replica.cc:330] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6: stopping tablet replica
I20240502 07:58:51.938346 30122 raft_consensus.cc:2229] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:51.938500 30122 raft_consensus.cc:2258] T 7343c21e7dfe47a2bfcc793de456d971 P 7feef95690fe4b0bb736a884af9569e6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:51.939050 30122 tablet_replica.cc:330] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6: stopping tablet replica
I20240502 07:58:51.939126 30122 raft_consensus.cc:2229] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [term 1 LEADER]: Raft consensus shutting down.
I20240502 07:58:51.939309 30122 raft_consensus.cc:2258] T 862e4df4c8004aa0973a06875b587e5f P 7feef95690fe4b0bb736a884af9569e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:51.952908 30122 tablet_server.cc:194] TabletServer@127.29.106.131:0 shutdown complete.
I20240502 07:58:51.955960 30122 tablet_server.cc:178] TabletServer@127.29.106.132:0 shutting down...
I20240502 07:58:51.961750 30122 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 07:58:51.961990 30122 tablet_replica.cc:330] T 862e4df4c8004aa0973a06875b587e5f P e11ea6ca87794bbc8858fe02903eb3bb: stopping tablet replica
I20240502 07:58:51.962132 30122 raft_consensus.cc:2229] T 862e4df4c8004aa0973a06875b587e5f P e11ea6ca87794bbc8858fe02903eb3bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:51.962294 30122 raft_consensus.cc:2258] T 862e4df4c8004aa0973a06875b587e5f P e11ea6ca87794bbc8858fe02903eb3bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:51.975548 30122 tablet_server.cc:194] TabletServer@127.29.106.132:0 shutdown complete.
I20240502 07:58:51.977874 30122 master.cc:537] Master@127.29.106.190:40157 shutting down...
I20240502 07:58:51.984812 30122 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3dd356a7497f48efb4ba5523cf97dc50 [term 1 LEADER]: Raft consensus shutting down.
I20240502 07:58:51.985036 30122 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3dd356a7497f48efb4ba5523cf97dc50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:51.985136 30122 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3dd356a7497f48efb4ba5523cf97dc50: stopping tablet replica
I20240502 07:58:51.998562 30122 master.cc:552] Master@127.29.106.190:40157 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (26705 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20240502 07:58:52.007161 30122 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.106.190:42073
I20240502 07:58:52.007692 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 07:58:52.010871 30122 server_base.cc:1034] running on GCE node
W20240502 07:58:52.011421 32239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:52.012091 32241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:52.012972 32238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:52.013262 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:58:52.013314 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:58:52.013331 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636732013331 us; error 0 us; skew 500 ppm
I20240502 07:58:52.019032 30122 webserver.cc:460] Webserver started at http://127.29.106.190:34751/ using document root <none> and password file <none>
I20240502 07:58:52.019227 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:52.019281 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:52.019351 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:52.019829 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714636688849691-30122-0/minicluster-data/master-0-root/instance:
uuid: "698851a4fb1442c6b77697546e435425"
format_stamp: "Formatted at 2024-05-02 07:58:52 on dist-test-slave-m6l2"
I20240502 07:58:52.021663 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240502 07:58:52.022785 32246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:52.023150 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:52.023232 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714636688849691-30122-0/minicluster-data/master-0-root
uuid: "698851a4fb1442c6b77697546e435425"
format_stamp: "Formatted at 2024-05-02 07:58:52 on dist-test-slave-m6l2"
I20240502 07:58:52.023320 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714636688849691-30122-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714636688849691-30122-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714636688849691-30122-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:52.031450 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:52.031908 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:52.037314 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.190:42073
I20240502 07:58:52.040547 32298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.190:42073 every 8 connection(s)
I20240502 07:58:52.041038 32299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:52.042933 32299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 698851a4fb1442c6b77697546e435425: Bootstrap starting.
I20240502 07:58:52.043720 32299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 698851a4fb1442c6b77697546e435425: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:52.045513 32299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 698851a4fb1442c6b77697546e435425: No bootstrap required, opened a new log
I20240502 07:58:52.046074 32299 raft_consensus.cc:348] T 00000000000000000000000000000000 P 698851a4fb1442c6b77697546e435425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698851a4fb1442c6b77697546e435425" member_type: VOTER }
I20240502 07:58:52.046187 32299 raft_consensus.cc:374] T 00000000000000000000000000000000 P 698851a4fb1442c6b77697546e435425 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:52.046211 32299 raft_consensus.cc:729] T 00000000000000000000000000000000 P 698851a4fb1442c6b77697546e435425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 698851a4fb1442c6b77697546e435425, State: Initialized, Role: FOLLOWER
I20240502 07:58:52.046335 32299 consensus_queue.cc:260] T 00000000000000000000000000000000 P 698851a4fb1442c6b77697546e435425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698851a4fb1442c6b77697546e435425" member_type: VOTER }
I20240502 07:58:52.046419 32299 raft_consensus.cc:388] T 00000000000000000000000000000000 P 698851a4fb1442c6b77697546e435425 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:58:52.046444 32299 raft_consensus.cc:482] T 00000000000000000000000000000000 P 698851a4fb1442c6b77697546e435425 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:58:52.046504 32299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 698851a4fb1442c6b77697546e435425 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:52.047346 32299 raft_consensus.cc:504] T 00000000000000000000000000000000 P 698851a4fb1442c6b77697546e435425 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698851a4fb1442c6b77697546e435425" member_type: VOTER }
I20240502 07:58:52.047462 32299 leader_election.cc:302] T 00000000000000000000000000000000 P 698851a4fb1442c6b77697546e435425 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 698851a4fb1442c6b77697546e435425; no voters:
I20240502 07:58:52.047673 32299 leader_election.cc:288] T 00000000000000000000000000000000 P 698851a4fb1442c6b77697546e435425 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:58:52.047878 32302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 698851a4fb1442c6b77697546e435425 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:52.048157 32302 raft_consensus.cc:686] T 00000000000000000000000000000000 P 698851a4fb1442c6b77697546e435425 [term 1 LEADER]: Becoming Leader. State: Replica: 698851a4fb1442c6b77697546e435425, State: Running, Role: LEADER
I20240502 07:58:52.048380 32299 sys_catalog.cc:564] T 00000000000000000000000000000000 P 698851a4fb1442c6b77697546e435425 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 07:58:52.048413 32302 consensus_queue.cc:237] T 00000000000000000000000000000000 P 698851a4fb1442c6b77697546e435425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698851a4fb1442c6b77697546e435425" member_type: VOTER }
I20240502 07:58:52.049420 32304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 698851a4fb1442c6b77697546e435425 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 698851a4fb1442c6b77697546e435425. Latest consensus state: current_term: 1 leader_uuid: "698851a4fb1442c6b77697546e435425" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698851a4fb1442c6b77697546e435425" member_type: VOTER } }
I20240502 07:58:52.049547 32304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 698851a4fb1442c6b77697546e435425 [sys.catalog]: This master's current role is: LEADER
I20240502 07:58:52.050084 32311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 07:58:52.049969 32303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 698851a4fb1442c6b77697546e435425 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "698851a4fb1442c6b77697546e435425" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698851a4fb1442c6b77697546e435425" member_type: VOTER } }
I20240502 07:58:52.050271 32303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 698851a4fb1442c6b77697546e435425 [sys.catalog]: This master's current role is: LEADER
I20240502 07:58:52.051455 32311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 07:58:52.051729 30122 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 07:58:52.053970 32311 catalog_manager.cc:1347] Generated new cluster ID: 600c64e161df4a66bce6fca939d0d602
I20240502 07:58:52.054047 32311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 07:58:52.083595 32311 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 07:58:52.084239 32311 catalog_manager.cc:1504] Loading token signing keys...
I20240502 07:58:52.092234 32311 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 698851a4fb1442c6b77697546e435425: Generated new TSK 0
I20240502 07:58:52.092568 32311 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 07:58:52.116154 30122 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 8.0911e-05s
I20240502 07:59:04.196408 30122 master.cc:537] Master@127.29.106.190:42073 shutting down...
I20240502 07:59:04.199553 30122 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 698851a4fb1442c6b77697546e435425 [term 1 LEADER]: Raft consensus shutting down.
I20240502 07:59:04.199728 30122 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 698851a4fb1442c6b77697546e435425 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:04.199779 30122 tablet_replica.cc:330] T 00000000000000000000000000000000 P 698851a4fb1442c6b77697546e435425: stopping tablet replica
I20240502 07:59:04.212131 30122 master.cc:552] Master@127.29.106.190:42073 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12209 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20240502 07:59:04.216352 30122 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.106.190:39455
I20240502 07:59:04.216670 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:04.219550 32321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:04.220181 32324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:04.220896 30122 server_base.cc:1034] running on GCE node
W20240502 07:59:04.221704 32322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:04.221940 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:59:04.221992 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:59:04.222028 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636744222026 us; error 0 us; skew 500 ppm
I20240502 07:59:04.222999 30122 webserver.cc:460] Webserver started at http://127.29.106.190:34027/ using document root <none> and password file <none>
I20240502 07:59:04.223165 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:04.223225 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:04.223315 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:04.223734 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636688849691-30122-0/minicluster-data/master-0-root/instance:
uuid: "198a345a4699449a9b21777ac922aae5"
format_stamp: "Formatted at 2024-05-02 07:59:04 on dist-test-slave-m6l2"
I20240502 07:59:04.225481 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 07:59:04.226573 32329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:04.226883 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:04.227061 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636688849691-30122-0/minicluster-data/master-0-root
uuid: "198a345a4699449a9b21777ac922aae5"
format_stamp: "Formatted at 2024-05-02 07:59:04 on dist-test-slave-m6l2"
I20240502 07:59:04.227150 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636688849691-30122-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636688849691-30122-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636688849691-30122-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:04.245751 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:04.246140 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:04.250185 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.190:39455
I20240502 07:59:04.250641 32381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.190:39455 every 8 connection(s)
I20240502 07:59:04.251171 32382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:04.252946 32382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 198a345a4699449a9b21777ac922aae5: Bootstrap starting.
I20240502 07:59:04.253844 32382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 198a345a4699449a9b21777ac922aae5: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:04.254981 32382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 198a345a4699449a9b21777ac922aae5: No bootstrap required, opened a new log
I20240502 07:59:04.255344 32382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 198a345a4699449a9b21777ac922aae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198a345a4699449a9b21777ac922aae5" member_type: VOTER }
I20240502 07:59:04.255435 32382 raft_consensus.cc:374] T 00000000000000000000000000000000 P 198a345a4699449a9b21777ac922aae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:04.255460 32382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 198a345a4699449a9b21777ac922aae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 198a345a4699449a9b21777ac922aae5, State: Initialized, Role: FOLLOWER
I20240502 07:59:04.255581 32382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 198a345a4699449a9b21777ac922aae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198a345a4699449a9b21777ac922aae5" member_type: VOTER }
I20240502 07:59:04.255661 32382 raft_consensus.cc:388] T 00000000000000000000000000000000 P 198a345a4699449a9b21777ac922aae5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:59:04.255688 32382 raft_consensus.cc:482] T 00000000000000000000000000000000 P 198a345a4699449a9b21777ac922aae5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:59:04.255721 32382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 198a345a4699449a9b21777ac922aae5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:04.256448 32382 raft_consensus.cc:504] T 00000000000000000000000000000000 P 198a345a4699449a9b21777ac922aae5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198a345a4699449a9b21777ac922aae5" member_type: VOTER }
I20240502 07:59:04.256557 32382 leader_election.cc:302] T 00000000000000000000000000000000 P 198a345a4699449a9b21777ac922aae5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 198a345a4699449a9b21777ac922aae5; no voters:
I20240502 07:59:04.256707 32382 leader_election.cc:288] T 00000000000000000000000000000000 P 198a345a4699449a9b21777ac922aae5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:59:04.256887 32385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 198a345a4699449a9b21777ac922aae5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:04.257073 32385 raft_consensus.cc:686] T 00000000000000000000000000000000 P 198a345a4699449a9b21777ac922aae5 [term 1 LEADER]: Becoming Leader. State: Replica: 198a345a4699449a9b21777ac922aae5, State: Running, Role: LEADER
I20240502 07:59:04.257133 32382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 198a345a4699449a9b21777ac922aae5 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 07:59:04.257215 32385 consensus_queue.cc:237] T 00000000000000000000000000000000 P 198a345a4699449a9b21777ac922aae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198a345a4699449a9b21777ac922aae5" member_type: VOTER }
I20240502 07:59:04.257630 32387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198a345a4699449a9b21777ac922aae5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 198a345a4699449a9b21777ac922aae5. Latest consensus state: current_term: 1 leader_uuid: "198a345a4699449a9b21777ac922aae5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198a345a4699449a9b21777ac922aae5" member_type: VOTER } }
I20240502 07:59:04.257613 32386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198a345a4699449a9b21777ac922aae5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "198a345a4699449a9b21777ac922aae5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198a345a4699449a9b21777ac922aae5" member_type: VOTER } }
I20240502 07:59:04.257717 32387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198a345a4699449a9b21777ac922aae5 [sys.catalog]: This master's current role is: LEADER
I20240502 07:59:04.257737 32386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198a345a4699449a9b21777ac922aae5 [sys.catalog]: This master's current role is: LEADER
I20240502 07:59:04.257982 32390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 07:59:04.258785 32390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 07:59:04.259014 30122 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 07:59:04.260656 32390 catalog_manager.cc:1347] Generated new cluster ID: 7de294ff68ad496fafada7293543ff6a
I20240502 07:59:04.260718 32390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 07:59:04.269169 32390 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 07:59:04.269798 32390 catalog_manager.cc:1504] Loading token signing keys...
I20240502 07:59:04.279148 32390 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 198a345a4699449a9b21777ac922aae5: Generated new TSK 0
I20240502 07:59:04.279345 32390 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 07:59:04.291579 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:04.294567 32403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:04.295212 32406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:04.295836 32404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:04.296511 30122 server_base.cc:1034] running on GCE node
I20240502 07:59:04.296695 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:59:04.296739 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:59:04.296758 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636744296758 us; error 0 us; skew 500 ppm
I20240502 07:59:04.297938 30122 webserver.cc:460] Webserver started at http://127.29.106.129:38445/ using document root <none> and password file <none>
I20240502 07:59:04.298110 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:04.298166 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:04.298249 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:04.298691 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636688849691-30122-0/minicluster-data/ts-0-root/instance:
uuid: "420a1ff4fdd04407aa00cc956bda6bd2"
format_stamp: "Formatted at 2024-05-02 07:59:04 on dist-test-slave-m6l2"
I20240502 07:59:04.300396 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240502 07:59:04.301424 32411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:04.301713 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:04.301796 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636688849691-30122-0/minicluster-data/ts-0-root
uuid: "420a1ff4fdd04407aa00cc956bda6bd2"
format_stamp: "Formatted at 2024-05-02 07:59:04 on dist-test-slave-m6l2"
I20240502 07:59:04.301877 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636688849691-30122-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636688849691-30122-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636688849691-30122-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:04.309616 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:04.310065 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:04.310376 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:04.310895 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:04.310940 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:04.310993 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:04.311024 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:04.315642 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.129:39039
I20240502 07:59:04.315665 32474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.129:39039 every 8 connection(s)
I20240502 07:59:04.316627 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:04.319404 32479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:04.320019 32480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:04.320632 32482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:04.321365 30122 server_base.cc:1034] running on GCE node
I20240502 07:59:04.321537 32475 heartbeater.cc:347] Connected to a master server at 127.29.106.190:39455
I20240502 07:59:04.321657 32475 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:04.321727 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:59:04.321786 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:59:04.321821 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636744321820 us; error 0 us; skew 500 ppm
I20240502 07:59:04.321930 32475 heartbeater.cc:511] Master 127.29.106.190:39455 requested a full tablet report, sending...
I20240502 07:59:04.322619 32346 ts_manager.cc:194] Registered new tserver with Master: 420a1ff4fdd04407aa00cc956bda6bd2 (127.29.106.129:39039)
I20240502 07:59:04.323160 30122 webserver.cc:460] Webserver started at http://127.29.106.130:40111/ using document root <none> and password file <none>
I20240502 07:59:04.323329 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:04.323379 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:04.323453 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:04.323696 32346 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39664
I20240502 07:59:04.323848 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636688849691-30122-0/minicluster-data/ts-1-root/instance:
uuid: "18718561adcd4095b42a1174be1df323"
format_stamp: "Formatted at 2024-05-02 07:59:04 on dist-test-slave-m6l2"
I20240502 07:59:04.325496 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 07:59:04.326480 32487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:04.326721 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:04.326784 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636688849691-30122-0/minicluster-data/ts-1-root
uuid: "18718561adcd4095b42a1174be1df323"
format_stamp: "Formatted at 2024-05-02 07:59:04 on dist-test-slave-m6l2"
I20240502 07:59:04.326855 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636688849691-30122-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636688849691-30122-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636688849691-30122-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:04.349197 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:04.349642 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:04.349937 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:04.350395 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:04.350433 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:04.350481 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:04.350512 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:04.354992 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.130:33015
I20240502 07:59:04.355211 32550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.130:33015 every 8 connection(s)
I20240502 07:59:04.355971 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:04.358911 32555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:04.359556 32554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:04.360214 32557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:04.360826 32551 heartbeater.cc:347] Connected to a master server at 127.29.106.190:39455
I20240502 07:59:04.360929 32551 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:04.361187 32551 heartbeater.cc:511] Master 127.29.106.190:39455 requested a full tablet report, sending...
I20240502 07:59:04.361511 30122 server_base.cc:1034] running on GCE node
I20240502 07:59:04.361706 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:59:04.361759 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:59:04.361780 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636744361780 us; error 0 us; skew 500 ppm
I20240502 07:59:04.361872 32346 ts_manager.cc:194] Registered new tserver with Master: 18718561adcd4095b42a1174be1df323 (127.29.106.130:33015)
I20240502 07:59:04.362507 32346 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39670
I20240502 07:59:04.362821 30122 webserver.cc:460] Webserver started at http://127.29.106.131:38869/ using document root <none> and password file <none>
I20240502 07:59:04.363014 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:04.363065 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:04.363145 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:04.363540 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636688849691-30122-0/minicluster-data/ts-2-root/instance:
uuid: "ac393bc733b94cc19bf4bc3c40300ef3"
format_stamp: "Formatted at 2024-05-02 07:59:04 on dist-test-slave-m6l2"
I20240502 07:59:04.365029 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 07:59:04.365981 32562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:04.366237 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:04.366302 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636688849691-30122-0/minicluster-data/ts-2-root
uuid: "ac393bc733b94cc19bf4bc3c40300ef3"
format_stamp: "Formatted at 2024-05-02 07:59:04 on dist-test-slave-m6l2"
I20240502 07:59:04.366370 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636688849691-30122-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636688849691-30122-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636688849691-30122-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:04.397714 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:04.398154 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:04.398453 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:04.398896 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:04.398936 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:04.398980 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:04.399012 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:04.403748 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.131:36609
I20240502 07:59:04.403848 32625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.131:36609 every 8 connection(s)
I20240502 07:59:04.408396 32626 heartbeater.cc:347] Connected to a master server at 127.29.106.190:39455
I20240502 07:59:04.408507 32626 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:04.408739 32626 heartbeater.cc:511] Master 127.29.106.190:39455 requested a full tablet report, sending...
I20240502 07:59:04.409364 32346 ts_manager.cc:194] Registered new tserver with Master: ac393bc733b94cc19bf4bc3c40300ef3 (127.29.106.131:36609)
I20240502 07:59:04.409816 30122 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005642129s
I20240502 07:59:04.410012 32346 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39684
I20240502 07:59:05.324827 32475 heartbeater.cc:503] Master 127.29.106.190:39455 was elected leader, sending a full tablet report...
I20240502 07:59:05.363624 32551 heartbeater.cc:503] Master 127.29.106.190:39455 was elected leader, sending a full tablet report...
I20240502 07:59:05.411044 32626 heartbeater.cc:503] Master 127.29.106.190:39455 was elected leader, sending a full tablet report...
I20240502 07:59:06.259280 32400 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:59:08.259718 32400 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:59:10.260161 32400 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:59:12.260682 32400 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:59:14.261127 32400 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:59:16.261596 32400 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:59:16.490424 30122 tablet_server.cc:178] TabletServer@127.29.106.129:0 shutting down...
I20240502 07:59:16.495374 30122 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 07:59:16.508416 30122 tablet_server.cc:194] TabletServer@127.29.106.129:0 shutdown complete.
I20240502 07:59:16.510854 30122 tablet_server.cc:178] TabletServer@127.29.106.130:0 shutting down...
I20240502 07:59:16.515748 30122 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 07:59:16.528110 30122 tablet_server.cc:194] TabletServer@127.29.106.130:0 shutdown complete.
I20240502 07:59:16.530562 30122 tablet_server.cc:178] TabletServer@127.29.106.131:0 shutting down...
I20240502 07:59:16.535418 30122 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 07:59:16.547973 30122 tablet_server.cc:194] TabletServer@127.29.106.131:0 shutdown complete.
I20240502 07:59:16.549950 30122 master.cc:537] Master@127.29.106.190:39455 shutting down...
I20240502 07:59:16.554788 30122 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 198a345a4699449a9b21777ac922aae5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 07:59:16.555038 30122 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 198a345a4699449a9b21777ac922aae5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:16.555112 30122 tablet_replica.cc:330] T 00000000000000000000000000000000 P 198a345a4699449a9b21777ac922aae5: stopping tablet replica
I20240502 07:59:16.569059 30122 master.cc:552] Master@127.29.106.190:39455 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12358 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20240502 07:59:16.575093 30122 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.106.190:33371
I20240502 07:59:16.575441 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:16.578578 32631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:16.579485 32630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:16.580410 32633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:16.580981 30122 server_base.cc:1034] running on GCE node
I20240502 07:59:16.581203 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:59:16.581250 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:59:16.581269 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636756581269 us; error 0 us; skew 500 ppm
I20240502 07:59:16.582437 30122 webserver.cc:460] Webserver started at http://127.29.106.190:34745/ using document root <none> and password file <none>
I20240502 07:59:16.582623 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:16.582691 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:16.582789 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:16.583323 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636688849691-30122-0/minicluster-data/master-0-root/instance:
uuid: "92f8972ca9594c0ebbdfc3fdf6d415bb"
format_stamp: "Formatted at 2024-05-02 07:59:16 on dist-test-slave-m6l2"
I20240502 07:59:16.585292 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:59:16.586402 32638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:16.586730 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:16.586809 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636688849691-30122-0/minicluster-data/master-0-root
uuid: "92f8972ca9594c0ebbdfc3fdf6d415bb"
format_stamp: "Formatted at 2024-05-02 07:59:16 on dist-test-slave-m6l2"
I20240502 07:59:16.586895 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636688849691-30122-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636688849691-30122-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636688849691-30122-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:16.595443 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:16.595901 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:16.600993 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.190:33371
I20240502 07:59:16.602715 32690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.190:33371 every 8 connection(s)
I20240502 07:59:16.603219 32691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:16.605406 32691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92f8972ca9594c0ebbdfc3fdf6d415bb: Bootstrap starting.
I20240502 07:59:16.606266 32691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92f8972ca9594c0ebbdfc3fdf6d415bb: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:16.607525 32691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92f8972ca9594c0ebbdfc3fdf6d415bb: No bootstrap required, opened a new log
I20240502 07:59:16.607933 32691 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92f8972ca9594c0ebbdfc3fdf6d415bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92f8972ca9594c0ebbdfc3fdf6d415bb" member_type: VOTER }
I20240502 07:59:16.608032 32691 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92f8972ca9594c0ebbdfc3fdf6d415bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:16.608054 32691 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92f8972ca9594c0ebbdfc3fdf6d415bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f8972ca9594c0ebbdfc3fdf6d415bb, State: Initialized, Role: FOLLOWER
I20240502 07:59:16.608175 32691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92f8972ca9594c0ebbdfc3fdf6d415bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92f8972ca9594c0ebbdfc3fdf6d415bb" member_type: VOTER }
I20240502 07:59:16.608229 32691 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92f8972ca9594c0ebbdfc3fdf6d415bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:59:16.608284 32691 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92f8972ca9594c0ebbdfc3fdf6d415bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:59:16.608343 32691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92f8972ca9594c0ebbdfc3fdf6d415bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:16.609050 32691 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92f8972ca9594c0ebbdfc3fdf6d415bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92f8972ca9594c0ebbdfc3fdf6d415bb" member_type: VOTER }
I20240502 07:59:16.609169 32691 leader_election.cc:302] T 00000000000000000000000000000000 P 92f8972ca9594c0ebbdfc3fdf6d415bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92f8972ca9594c0ebbdfc3fdf6d415bb; no voters:
I20240502 07:59:16.609340 32691 leader_election.cc:288] T 00000000000000000000000000000000 P 92f8972ca9594c0ebbdfc3fdf6d415bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:59:16.609529 32694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92f8972ca9594c0ebbdfc3fdf6d415bb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:16.609776 32691 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92f8972ca9594c0ebbdfc3fdf6d415bb [sys.catalog]: configured and running, proceeding with master startup.
I20240502 07:59:16.609799 32694 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92f8972ca9594c0ebbdfc3fdf6d415bb [term 1 LEADER]: Becoming Leader. State: Replica: 92f8972ca9594c0ebbdfc3fdf6d415bb, State: Running, Role: LEADER
I20240502 07:59:16.609987 32694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92f8972ca9594c0ebbdfc3fdf6d415bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92f8972ca9594c0ebbdfc3fdf6d415bb" member_type: VOTER }
I20240502 07:59:16.610546 32696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92f8972ca9594c0ebbdfc3fdf6d415bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92f8972ca9594c0ebbdfc3fdf6d415bb. Latest consensus state: current_term: 1 leader_uuid: "92f8972ca9594c0ebbdfc3fdf6d415bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92f8972ca9594c0ebbdfc3fdf6d415bb" member_type: VOTER } }
I20240502 07:59:16.610743 32696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92f8972ca9594c0ebbdfc3fdf6d415bb [sys.catalog]: This master's current role is: LEADER
I20240502 07:59:16.610596 32695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92f8972ca9594c0ebbdfc3fdf6d415bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92f8972ca9594c0ebbdfc3fdf6d415bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92f8972ca9594c0ebbdfc3fdf6d415bb" member_type: VOTER } }
I20240502 07:59:16.611023 32695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92f8972ca9594c0ebbdfc3fdf6d415bb [sys.catalog]: This master's current role is: LEADER
I20240502 07:59:16.611246 32703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 07:59:16.612114 32703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 07:59:16.612385 30122 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 07:59:16.614090 32703 catalog_manager.cc:1347] Generated new cluster ID: cf757160ed484dc5a8e6345f1894665c
I20240502 07:59:16.614162 32703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 07:59:16.628449 32703 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 07:59:16.629257 32703 catalog_manager.cc:1504] Loading token signing keys...
I20240502 07:59:16.633890 32703 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 92f8972ca9594c0ebbdfc3fdf6d415bb: Generated new TSK 0
I20240502 07:59:16.634089 32703 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 07:59:16.645277 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:16.648610 32712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:16.649201 32713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:16.649912 30122 server_base.cc:1034] running on GCE node
W20240502 07:59:16.650887 32715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:16.651142 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:59:16.651190 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:59:16.651207 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636756651207 us; error 0 us; skew 500 ppm
I20240502 07:59:16.652256 30122 webserver.cc:460] Webserver started at http://127.29.106.129:32915/ using document root <none> and password file <none>
I20240502 07:59:16.652455 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:16.652508 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:16.652570 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:16.652949 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636688849691-30122-0/minicluster-data/ts-0-root/instance:
uuid: "3925233c21294d01a8d3f3a545cb9f10"
format_stamp: "Formatted at 2024-05-02 07:59:16 on dist-test-slave-m6l2"
I20240502 07:59:16.654454 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 07:59:16.655491 32720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:16.655789 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:16.655855 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636688849691-30122-0/minicluster-data/ts-0-root
uuid: "3925233c21294d01a8d3f3a545cb9f10"
format_stamp: "Formatted at 2024-05-02 07:59:16 on dist-test-slave-m6l2"
I20240502 07:59:16.655915 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636688849691-30122-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636688849691-30122-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636688849691-30122-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:16.666796 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:16.667155 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:16.667414 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:16.667874 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:16.667913 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:16.667944 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:16.667960 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:16.672116 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.129:35503
I20240502 07:59:16.672151 315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.129:35503 every 8 connection(s)
I20240502 07:59:16.672919 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:16.675801 320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:16.676393 321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:16.677013 323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:16.677470 316 heartbeater.cc:347] Connected to a master server at 127.29.106.190:33371
I20240502 07:59:16.677551 316 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:16.677747 316 heartbeater.cc:511] Master 127.29.106.190:33371 requested a full tablet report, sending...
I20240502 07:59:16.678041 30122 server_base.cc:1034] running on GCE node
I20240502 07:59:16.678195 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:59:16.678265 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:59:16.678298 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636756678297 us; error 0 us; skew 500 ppm
I20240502 07:59:16.678457 32655 ts_manager.cc:194] Registered new tserver with Master: 3925233c21294d01a8d3f3a545cb9f10 (127.29.106.129:35503)
I20240502 07:59:16.679337 30122 webserver.cc:460] Webserver started at http://127.29.106.130:38347/ using document root <none> and password file <none>
I20240502 07:59:16.679405 32655 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33696
I20240502 07:59:16.679491 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:16.679548 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:16.679622 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:16.680078 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636688849691-30122-0/minicluster-data/ts-1-root/instance:
uuid: "50f01cef91044654a482b5369ab12746"
format_stamp: "Formatted at 2024-05-02 07:59:16 on dist-test-slave-m6l2"
I20240502 07:59:16.681510 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:16.682552 328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:16.682853 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:16.682919 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636688849691-30122-0/minicluster-data/ts-1-root
uuid: "50f01cef91044654a482b5369ab12746"
format_stamp: "Formatted at 2024-05-02 07:59:16 on dist-test-slave-m6l2"
I20240502 07:59:16.683027 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636688849691-30122-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636688849691-30122-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636688849691-30122-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:16.716846 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:16.717260 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:16.717548 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:16.717985 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:16.718029 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:16.718066 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:16.718097 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:16.722641 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.130:39569
I20240502 07:59:16.722771 391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.130:39569 every 8 connection(s)
I20240502 07:59:16.723562 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:16.726796 396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:16.727604 398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:16.727697 392 heartbeater.cc:347] Connected to a master server at 127.29.106.190:33371
I20240502 07:59:16.727816 392 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:16.728049 392 heartbeater.cc:511] Master 127.29.106.190:33371 requested a full tablet report, sending...
W20240502 07:59:16.728466 395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:16.728906 32655 ts_manager.cc:194] Registered new tserver with Master: 50f01cef91044654a482b5369ab12746 (127.29.106.130:39569)
I20240502 07:59:16.728955 30122 server_base.cc:1034] running on GCE node
I20240502 07:59:16.729130 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:59:16.729176 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:59:16.729193 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636756729193 us; error 0 us; skew 500 ppm
I20240502 07:59:16.729486 32655 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33710
I20240502 07:59:16.730232 30122 webserver.cc:460] Webserver started at http://127.29.106.131:38973/ using document root <none> and password file <none>
I20240502 07:59:16.730378 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:16.730422 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:16.730479 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:16.730837 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636688849691-30122-0/minicluster-data/ts-2-root/instance:
uuid: "ed62331907cf4f7988c03c2e16a8af6e"
format_stamp: "Formatted at 2024-05-02 07:59:16 on dist-test-slave-m6l2"
I20240502 07:59:16.732477 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240502 07:59:16.733484 403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:16.733793 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:16.733860 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636688849691-30122-0/minicluster-data/ts-2-root
uuid: "ed62331907cf4f7988c03c2e16a8af6e"
format_stamp: "Formatted at 2024-05-02 07:59:16 on dist-test-slave-m6l2"
I20240502 07:59:16.733934 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636688849691-30122-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636688849691-30122-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636688849691-30122-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:16.741400 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:16.741818 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:16.742110 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:16.742561 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:16.742604 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:16.742655 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:16.742686 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:16.747468 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.131:42163
I20240502 07:59:16.747505 466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.131:42163 every 8 connection(s)
I20240502 07:59:16.748473 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:16.751571 471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:16.752422 470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:16.752849 467 heartbeater.cc:347] Connected to a master server at 127.29.106.190:33371
W20240502 07:59:16.752945 473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:16.752955 467 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:16.753258 467 heartbeater.cc:511] Master 127.29.106.190:33371 requested a full tablet report, sending...
I20240502 07:59:16.754035 32655 ts_manager.cc:194] Registered new tserver with Master: ed62331907cf4f7988c03c2e16a8af6e (127.29.106.131:42163)
I20240502 07:59:16.754097 30122 server_base.cc:1034] running on GCE node
I20240502 07:59:16.754329 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:59:16.754372 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:59:16.754388 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636756754388 us; error 0 us; skew 500 ppm
I20240502 07:59:16.754688 32655 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33712
I20240502 07:59:16.755439 30122 webserver.cc:460] Webserver started at http://127.29.106.132:33717/ using document root <none> and password file <none>
I20240502 07:59:16.755594 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:16.755645 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:16.755705 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:16.756117 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636688849691-30122-0/minicluster-data/ts-3-root/instance:
uuid: "d672f2b2f39e4343b81540a8e04d99d2"
format_stamp: "Formatted at 2024-05-02 07:59:16 on dist-test-slave-m6l2"
I20240502 07:59:16.757894 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:16.758955 478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:16.759335 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:16.759405 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636688849691-30122-0/minicluster-data/ts-3-root
uuid: "d672f2b2f39e4343b81540a8e04d99d2"
format_stamp: "Formatted at 2024-05-02 07:59:16 on dist-test-slave-m6l2"
I20240502 07:59:16.759482 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636688849691-30122-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636688849691-30122-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636688849691-30122-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:16.771191 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:16.771632 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:16.771910 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:16.772431 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:16.772473 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:16.772524 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:16.772557 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:16.777107 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.132:37309
I20240502 07:59:16.777184 541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.132:37309 every 8 connection(s)
I20240502 07:59:16.782012 542 heartbeater.cc:347] Connected to a master server at 127.29.106.190:33371
I20240502 07:59:16.782133 542 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:16.782356 542 heartbeater.cc:511] Master 127.29.106.190:33371 requested a full tablet report, sending...
I20240502 07:59:16.783057 32655 ts_manager.cc:194] Registered new tserver with Master: d672f2b2f39e4343b81540a8e04d99d2 (127.29.106.132:37309)
I20240502 07:59:16.783282 30122 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.005759232s
I20240502 07:59:16.783603 32655 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33724
I20240502 07:59:17.680485 316 heartbeater.cc:503] Master 127.29.106.190:33371 was elected leader, sending a full tablet report...
I20240502 07:59:17.730544 392 heartbeater.cc:503] Master 127.29.106.190:33371 was elected leader, sending a full tablet report...
I20240502 07:59:17.755766 467 heartbeater.cc:503] Master 127.29.106.190:33371 was elected leader, sending a full tablet report...
I20240502 07:59:17.784627 542 heartbeater.cc:503] Master 127.29.106.190:33371 was elected leader, sending a full tablet report...
I20240502 07:59:17.816541 30122 test_util.cc:268] Using random seed: -86092583
I20240502 07:59:17.825229 32655 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33746:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 07:59:17.836474 506 tablet_service.cc:1467] Processing CreateTablet for tablet 336d25b6bbab4349a857d25763123c8e (DEFAULT_TABLE table=test-workload [id=62dd8424d20644438f6663c898c175b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:17.836665 431 tablet_service.cc:1467] Processing CreateTablet for tablet 336d25b6bbab4349a857d25763123c8e (DEFAULT_TABLE table=test-workload [id=62dd8424d20644438f6663c898c175b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:17.836923 506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 336d25b6bbab4349a857d25763123c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:17.837003 431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 336d25b6bbab4349a857d25763123c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:17.837196 356 tablet_service.cc:1467] Processing CreateTablet for tablet 336d25b6bbab4349a857d25763123c8e (DEFAULT_TABLE table=test-workload [id=62dd8424d20644438f6663c898c175b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:17.837445 356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 336d25b6bbab4349a857d25763123c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:17.839681 563 tablet_bootstrap.cc:492] T 336d25b6bbab4349a857d25763123c8e P d672f2b2f39e4343b81540a8e04d99d2: Bootstrap starting.
I20240502 07:59:17.840044 565 tablet_bootstrap.cc:492] T 336d25b6bbab4349a857d25763123c8e P ed62331907cf4f7988c03c2e16a8af6e: Bootstrap starting.
I20240502 07:59:17.840490 564 tablet_bootstrap.cc:492] T 336d25b6bbab4349a857d25763123c8e P 50f01cef91044654a482b5369ab12746: Bootstrap starting.
I20240502 07:59:17.841028 563 tablet_bootstrap.cc:654] T 336d25b6bbab4349a857d25763123c8e P d672f2b2f39e4343b81540a8e04d99d2: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:17.841028 565 tablet_bootstrap.cc:654] T 336d25b6bbab4349a857d25763123c8e P ed62331907cf4f7988c03c2e16a8af6e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:17.841606 564 tablet_bootstrap.cc:654] T 336d25b6bbab4349a857d25763123c8e P 50f01cef91044654a482b5369ab12746: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:17.842522 563 tablet_bootstrap.cc:492] T 336d25b6bbab4349a857d25763123c8e P d672f2b2f39e4343b81540a8e04d99d2: No bootstrap required, opened a new log
I20240502 07:59:17.842617 565 tablet_bootstrap.cc:492] T 336d25b6bbab4349a857d25763123c8e P ed62331907cf4f7988c03c2e16a8af6e: No bootstrap required, opened a new log
I20240502 07:59:17.842626 563 ts_tablet_manager.cc:1367] T 336d25b6bbab4349a857d25763123c8e P d672f2b2f39e4343b81540a8e04d99d2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:17.842748 565 ts_tablet_manager.cc:1367] T 336d25b6bbab4349a857d25763123c8e P ed62331907cf4f7988c03c2e16a8af6e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:17.842720 564 tablet_bootstrap.cc:492] T 336d25b6bbab4349a857d25763123c8e P 50f01cef91044654a482b5369ab12746: No bootstrap required, opened a new log
I20240502 07:59:17.842913 564 ts_tablet_manager.cc:1367] T 336d25b6bbab4349a857d25763123c8e P 50f01cef91044654a482b5369ab12746: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:17.843268 563 raft_consensus.cc:348] T 336d25b6bbab4349a857d25763123c8e P d672f2b2f39e4343b81540a8e04d99d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d672f2b2f39e4343b81540a8e04d99d2" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 37309 } } peers { permanent_uuid: "ed62331907cf4f7988c03c2e16a8af6e" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 42163 } } peers { permanent_uuid: "50f01cef91044654a482b5369ab12746" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39569 } }
I20240502 07:59:17.843386 563 raft_consensus.cc:374] T 336d25b6bbab4349a857d25763123c8e P d672f2b2f39e4343b81540a8e04d99d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:17.843425 563 raft_consensus.cc:729] T 336d25b6bbab4349a857d25763123c8e P d672f2b2f39e4343b81540a8e04d99d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d672f2b2f39e4343b81540a8e04d99d2, State: Initialized, Role: FOLLOWER
I20240502 07:59:17.843456 565 raft_consensus.cc:348] T 336d25b6bbab4349a857d25763123c8e P ed62331907cf4f7988c03c2e16a8af6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d672f2b2f39e4343b81540a8e04d99d2" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 37309 } } peers { permanent_uuid: "ed62331907cf4f7988c03c2e16a8af6e" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 42163 } } peers { permanent_uuid: "50f01cef91044654a482b5369ab12746" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39569 } }
I20240502 07:59:17.843326 564 raft_consensus.cc:348] T 336d25b6bbab4349a857d25763123c8e P 50f01cef91044654a482b5369ab12746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d672f2b2f39e4343b81540a8e04d99d2" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 37309 } } peers { permanent_uuid: "ed62331907cf4f7988c03c2e16a8af6e" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 42163 } } peers { permanent_uuid: "50f01cef91044654a482b5369ab12746" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39569 } }
I20240502 07:59:17.843621 563 consensus_queue.cc:260] T 336d25b6bbab4349a857d25763123c8e P d672f2b2f39e4343b81540a8e04d99d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d672f2b2f39e4343b81540a8e04d99d2" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 37309 } } peers { permanent_uuid: "ed62331907cf4f7988c03c2e16a8af6e" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 42163 } } peers { permanent_uuid: "50f01cef91044654a482b5369ab12746" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39569 } }
I20240502 07:59:17.843731 564 raft_consensus.cc:374] T 336d25b6bbab4349a857d25763123c8e P 50f01cef91044654a482b5369ab12746 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:17.843823 564 raft_consensus.cc:729] T 336d25b6bbab4349a857d25763123c8e P 50f01cef91044654a482b5369ab12746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50f01cef91044654a482b5369ab12746, State: Initialized, Role: FOLLOWER
I20240502 07:59:17.843940 565 raft_consensus.cc:374] T 336d25b6bbab4349a857d25763123c8e P ed62331907cf4f7988c03c2e16a8af6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:17.843979 565 raft_consensus.cc:729] T 336d25b6bbab4349a857d25763123c8e P ed62331907cf4f7988c03c2e16a8af6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed62331907cf4f7988c03c2e16a8af6e, State: Initialized, Role: FOLLOWER
I20240502 07:59:17.844007 564 consensus_queue.cc:260] T 336d25b6bbab4349a857d25763123c8e P 50f01cef91044654a482b5369ab12746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d672f2b2f39e4343b81540a8e04d99d2" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 37309 } } peers { permanent_uuid: "ed62331907cf4f7988c03c2e16a8af6e" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 42163 } } peers { permanent_uuid: "50f01cef91044654a482b5369ab12746" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39569 } }
I20240502 07:59:17.844089 565 consensus_queue.cc:260] T 336d25b6bbab4349a857d25763123c8e P ed62331907cf4f7988c03c2e16a8af6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d672f2b2f39e4343b81540a8e04d99d2" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 37309 } } peers { permanent_uuid: "ed62331907cf4f7988c03c2e16a8af6e" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 42163 } } peers { permanent_uuid: "50f01cef91044654a482b5369ab12746" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39569 } }
I20240502 07:59:17.844139 563 ts_tablet_manager.cc:1398] T 336d25b6bbab4349a857d25763123c8e P d672f2b2f39e4343b81540a8e04d99d2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:17.844492 564 ts_tablet_manager.cc:1398] T 336d25b6bbab4349a857d25763123c8e P 50f01cef91044654a482b5369ab12746: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:17.844492 565 ts_tablet_manager.cc:1398] T 336d25b6bbab4349a857d25763123c8e P ed62331907cf4f7988c03c2e16a8af6e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240502 07:59:17.974565 393 tablet.cc:2369] T 336d25b6bbab4349a857d25763123c8e P 50f01cef91044654a482b5369ab12746: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:59:18.028851 570 raft_consensus.cc:482] T 336d25b6bbab4349a857d25763123c8e P 50f01cef91044654a482b5369ab12746 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:18.029104 570 raft_consensus.cc:504] T 336d25b6bbab4349a857d25763123c8e P 50f01cef91044654a482b5369ab12746 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d672f2b2f39e4343b81540a8e04d99d2" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 37309 } } peers { permanent_uuid: "ed62331907cf4f7988c03c2e16a8af6e" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 42163 } } peers { permanent_uuid: "50f01cef91044654a482b5369ab12746" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39569 } }
I20240502 07:59:18.029798 570 leader_election.cc:288] T 336d25b6bbab4349a857d25763123c8e P 50f01cef91044654a482b5369ab12746 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d672f2b2f39e4343b81540a8e04d99d2 (127.29.106.132:37309), ed62331907cf4f7988c03c2e16a8af6e (127.29.106.131:42163)
I20240502 07:59:18.033635 515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "336d25b6bbab4349a857d25763123c8e" candidate_uuid: "50f01cef91044654a482b5369ab12746" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d672f2b2f39e4343b81540a8e04d99d2" is_pre_election: true
I20240502 07:59:18.033790 441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "336d25b6bbab4349a857d25763123c8e" candidate_uuid: "50f01cef91044654a482b5369ab12746" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed62331907cf4f7988c03c2e16a8af6e" is_pre_election: true
I20240502 07:59:18.033805 515 raft_consensus.cc:2454] T 336d25b6bbab4349a857d25763123c8e P d672f2b2f39e4343b81540a8e04d99d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50f01cef91044654a482b5369ab12746 in term 0.
I20240502 07:59:18.034015 441 raft_consensus.cc:2454] T 336d25b6bbab4349a857d25763123c8e P ed62331907cf4f7988c03c2e16a8af6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50f01cef91044654a482b5369ab12746 in term 0.
I20240502 07:59:18.034260 332 leader_election.cc:302] T 336d25b6bbab4349a857d25763123c8e P 50f01cef91044654a482b5369ab12746 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50f01cef91044654a482b5369ab12746, d672f2b2f39e4343b81540a8e04d99d2; no voters:
I20240502 07:59:18.034473 570 raft_consensus.cc:2781] T 336d25b6bbab4349a857d25763123c8e P 50f01cef91044654a482b5369ab12746 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:18.034543 570 raft_consensus.cc:482] T 336d25b6bbab4349a857d25763123c8e P 50f01cef91044654a482b5369ab12746 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:18.034590 570 raft_consensus.cc:3037] T 336d25b6bbab4349a857d25763123c8e P 50f01cef91044654a482b5369ab12746 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:18.035713 570 raft_consensus.cc:504] T 336d25b6bbab4349a857d25763123c8e P 50f01cef91044654a482b5369ab12746 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d672f2b2f39e4343b81540a8e04d99d2" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 37309 } } peers { permanent_uuid: "ed62331907cf4f7988c03c2e16a8af6e" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 42163 } } peers { permanent_uuid: "50f01cef91044654a482b5369ab12746" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39569 } }
I20240502 07:59:18.036104 570 leader_election.cc:288] T 336d25b6bbab4349a857d25763123c8e P 50f01cef91044654a482b5369ab12746 [CANDIDATE]: Term 1 election: Requested vote from peers d672f2b2f39e4343b81540a8e04d99d2 (127.29.106.132:37309), ed62331907cf4f7988c03c2e16a8af6e (127.29.106.131:42163)
I20240502 07:59:18.036484 441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "336d25b6bbab4349a857d25763123c8e" candidate_uuid: "50f01cef91044654a482b5369ab12746" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed62331907cf4f7988c03c2e16a8af6e"
I20240502 07:59:18.036499 515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "336d25b6bbab4349a857d25763123c8e" candidate_uuid: "50f01cef91044654a482b5369ab12746" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d672f2b2f39e4343b81540a8e04d99d2"
I20240502 07:59:18.036623 441 raft_consensus.cc:3037] T 336d25b6bbab4349a857d25763123c8e P ed62331907cf4f7988c03c2e16a8af6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:18.036648 515 raft_consensus.cc:3037] T 336d25b6bbab4349a857d25763123c8e P d672f2b2f39e4343b81540a8e04d99d2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:18.037752 515 raft_consensus.cc:2454] T 336d25b6bbab4349a857d25763123c8e P d672f2b2f39e4343b81540a8e04d99d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50f01cef91044654a482b5369ab12746 in term 1.
I20240502 07:59:18.037847 441 raft_consensus.cc:2454] T 336d25b6bbab4349a857d25763123c8e P ed62331907cf4f7988c03c2e16a8af6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50f01cef91044654a482b5369ab12746 in term 1.
I20240502 07:59:18.038064 332 leader_election.cc:302] T 336d25b6bbab4349a857d25763123c8e P 50f01cef91044654a482b5369ab12746 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50f01cef91044654a482b5369ab12746, d672f2b2f39e4343b81540a8e04d99d2; no voters:
I20240502 07:59:18.038295 570 raft_consensus.cc:2781] T 336d25b6bbab4349a857d25763123c8e P 50f01cef91044654a482b5369ab12746 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:18.038486 570 raft_consensus.cc:686] T 336d25b6bbab4349a857d25763123c8e P 50f01cef91044654a482b5369ab12746 [term 1 LEADER]: Becoming Leader. State: Replica: 50f01cef91044654a482b5369ab12746, State: Running, Role: LEADER
I20240502 07:59:18.038619 570 consensus_queue.cc:237] T 336d25b6bbab4349a857d25763123c8e P 50f01cef91044654a482b5369ab12746 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d672f2b2f39e4343b81540a8e04d99d2" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 37309 } } peers { permanent_uuid: "ed62331907cf4f7988c03c2e16a8af6e" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 42163 } } peers { permanent_uuid: "50f01cef91044654a482b5369ab12746" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39569 } }
I20240502 07:59:18.040447 32653 catalog_manager.cc:5484] T 336d25b6bbab4349a857d25763123c8e P 50f01cef91044654a482b5369ab12746 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50f01cef91044654a482b5369ab12746 (127.29.106.130). New cstate: current_term: 1 leader_uuid: "50f01cef91044654a482b5369ab12746" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d672f2b2f39e4343b81540a8e04d99d2" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 37309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed62331907cf4f7988c03c2e16a8af6e" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 42163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50f01cef91044654a482b5369ab12746" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 39569 } health_report { overall_health: HEALTHY } } }
I20240502 07:59:18.468030 570 consensus_queue.cc:1035] T 336d25b6bbab4349a857d25763123c8e P 50f01cef91044654a482b5369ab12746 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d672f2b2f39e4343b81540a8e04d99d2" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 37309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:18.473860 574 consensus_queue.cc:1035] T 336d25b6bbab4349a857d25763123c8e P 50f01cef91044654a482b5369ab12746 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed62331907cf4f7988c03c2e16a8af6e" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:18.612227 32709 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:59:18.612676 32709 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 07:59:18.612746 32709 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:59:20.613101 32709 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:59:20.613466 32709 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 07:59:20.613526 32709 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:59:22.613888 32709 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:59:22.614320 32709 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 07:59:22.614388 32709 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:59:24.614740 32709 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:59:24.615180 32709 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 07:59:24.615247 32709 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:59:26.615650 32709 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:59:26.616094 32709 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 07:59:26.616168 32709 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:59:28.616524 32709 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:59:28.616873 32709 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 07:59:28.616930 32709 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:59:29.129132 30122 tablet_server.cc:178] TabletServer@127.29.106.129:0 shutting down...
I20240502 07:59:29.133675 30122 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 07:59:29.146443 30122 tablet_server.cc:194] TabletServer@127.29.106.129:0 shutdown complete.
I20240502 07:59:29.148833 30122 tablet_server.cc:178] TabletServer@127.29.106.130:0 shutting down...
I20240502 07:59:29.155437 30122 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 07:59:29.155676 30122 tablet_replica.cc:330] T 336d25b6bbab4349a857d25763123c8e P 50f01cef91044654a482b5369ab12746: stopping tablet replica
I20240502 07:59:29.155813 30122 raft_consensus.cc:2229] T 336d25b6bbab4349a857d25763123c8e P 50f01cef91044654a482b5369ab12746 [term 1 LEADER]: Raft consensus shutting down.
I20240502 07:59:29.156033 30122 raft_consensus.cc:2258] T 336d25b6bbab4349a857d25763123c8e P 50f01cef91044654a482b5369ab12746 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:29.169644 30122 tablet_server.cc:194] TabletServer@127.29.106.130:0 shutdown complete.
I20240502 07:59:29.171845 30122 tablet_server.cc:178] TabletServer@127.29.106.131:0 shutting down...
I20240502 07:59:29.177425 30122 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 07:59:29.177738 30122 tablet_replica.cc:330] T 336d25b6bbab4349a857d25763123c8e P ed62331907cf4f7988c03c2e16a8af6e: stopping tablet replica
I20240502 07:59:29.177877 30122 raft_consensus.cc:2229] T 336d25b6bbab4349a857d25763123c8e P ed62331907cf4f7988c03c2e16a8af6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:29.178005 30122 raft_consensus.cc:2258] T 336d25b6bbab4349a857d25763123c8e P ed62331907cf4f7988c03c2e16a8af6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:29.191022 30122 tablet_server.cc:194] TabletServer@127.29.106.131:0 shutdown complete.
I20240502 07:59:29.193754 30122 tablet_server.cc:178] TabletServer@127.29.106.132:0 shutting down...
I20240502 07:59:29.199196 30122 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 07:59:29.199463 30122 tablet_replica.cc:330] T 336d25b6bbab4349a857d25763123c8e P d672f2b2f39e4343b81540a8e04d99d2: stopping tablet replica
I20240502 07:59:29.199604 30122 raft_consensus.cc:2229] T 336d25b6bbab4349a857d25763123c8e P d672f2b2f39e4343b81540a8e04d99d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:29.199755 30122 raft_consensus.cc:2258] T 336d25b6bbab4349a857d25763123c8e P d672f2b2f39e4343b81540a8e04d99d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:29.213503 30122 tablet_server.cc:194] TabletServer@127.29.106.132:0 shutdown complete.
I20240502 07:59:29.215622 30122 master.cc:537] Master@127.29.106.190:33371 shutting down...
I20240502 07:59:29.219710 30122 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 92f8972ca9594c0ebbdfc3fdf6d415bb [term 1 LEADER]: Raft consensus shutting down.
I20240502 07:59:29.219852 30122 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 92f8972ca9594c0ebbdfc3fdf6d415bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:29.219904 30122 tablet_replica.cc:330] T 00000000000000000000000000000000 P 92f8972ca9594c0ebbdfc3fdf6d415bb: stopping tablet replica
I20240502 07:59:29.232547 30122 master.cc:552] Master@127.29.106.190:33371 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12664 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20240502 07:59:29.239697 30122 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.106.190:36227
I20240502 07:59:29.240003 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:29.242738 588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:29.243517 589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:29.244485 591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:29.245337 30122 server_base.cc:1034] running on GCE node
I20240502 07:59:29.245558 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:59:29.245601 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:59:29.245620 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636769245619 us; error 0 us; skew 500 ppm
I20240502 07:59:29.254863 30122 webserver.cc:460] Webserver started at http://127.29.106.190:38519/ using document root <none> and password file <none>
I20240502 07:59:29.255121 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:29.255189 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:29.255297 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:29.255817 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636688849691-30122-0/minicluster-data/master-0-root/instance:
uuid: "eae9ab4e6294455780b1f8ead3623326"
format_stamp: "Formatted at 2024-05-02 07:59:29 on dist-test-slave-m6l2"
I20240502 07:59:29.257663 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240502 07:59:29.258975 596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:29.259438 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:29.259531 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636688849691-30122-0/minicluster-data/master-0-root
uuid: "eae9ab4e6294455780b1f8ead3623326"
format_stamp: "Formatted at 2024-05-02 07:59:29 on dist-test-slave-m6l2"
I20240502 07:59:29.259619 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636688849691-30122-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636688849691-30122-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636688849691-30122-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:29.284478 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:29.284981 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:29.289839 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.190:36227
I20240502 07:59:29.292474 648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.190:36227 every 8 connection(s)
I20240502 07:59:29.292932 649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:29.294729 649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eae9ab4e6294455780b1f8ead3623326: Bootstrap starting.
I20240502 07:59:29.295460 649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eae9ab4e6294455780b1f8ead3623326: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:29.296700 649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eae9ab4e6294455780b1f8ead3623326: No bootstrap required, opened a new log
I20240502 07:59:29.297052 649 raft_consensus.cc:348] T 00000000000000000000000000000000 P eae9ab4e6294455780b1f8ead3623326 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae9ab4e6294455780b1f8ead3623326" member_type: VOTER }
I20240502 07:59:29.297140 649 raft_consensus.cc:374] T 00000000000000000000000000000000 P eae9ab4e6294455780b1f8ead3623326 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:29.297163 649 raft_consensus.cc:729] T 00000000000000000000000000000000 P eae9ab4e6294455780b1f8ead3623326 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae9ab4e6294455780b1f8ead3623326, State: Initialized, Role: FOLLOWER
I20240502 07:59:29.297295 649 consensus_queue.cc:260] T 00000000000000000000000000000000 P eae9ab4e6294455780b1f8ead3623326 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae9ab4e6294455780b1f8ead3623326" member_type: VOTER }
I20240502 07:59:29.297354 649 raft_consensus.cc:388] T 00000000000000000000000000000000 P eae9ab4e6294455780b1f8ead3623326 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:59:29.297384 649 raft_consensus.cc:482] T 00000000000000000000000000000000 P eae9ab4e6294455780b1f8ead3623326 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:59:29.297422 649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eae9ab4e6294455780b1f8ead3623326 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:29.298059 649 raft_consensus.cc:504] T 00000000000000000000000000000000 P eae9ab4e6294455780b1f8ead3623326 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae9ab4e6294455780b1f8ead3623326" member_type: VOTER }
I20240502 07:59:29.298166 649 leader_election.cc:302] T 00000000000000000000000000000000 P eae9ab4e6294455780b1f8ead3623326 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eae9ab4e6294455780b1f8ead3623326; no voters:
I20240502 07:59:29.298346 649 leader_election.cc:288] T 00000000000000000000000000000000 P eae9ab4e6294455780b1f8ead3623326 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:59:29.298508 652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eae9ab4e6294455780b1f8ead3623326 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:29.298749 652 raft_consensus.cc:686] T 00000000000000000000000000000000 P eae9ab4e6294455780b1f8ead3623326 [term 1 LEADER]: Becoming Leader. State: Replica: eae9ab4e6294455780b1f8ead3623326, State: Running, Role: LEADER
I20240502 07:59:29.298769 649 sys_catalog.cc:564] T 00000000000000000000000000000000 P eae9ab4e6294455780b1f8ead3623326 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 07:59:29.298909 652 consensus_queue.cc:237] T 00000000000000000000000000000000 P eae9ab4e6294455780b1f8ead3623326 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae9ab4e6294455780b1f8ead3623326" member_type: VOTER }
I20240502 07:59:29.299458 654 sys_catalog.cc:455] T 00000000000000000000000000000000 P eae9ab4e6294455780b1f8ead3623326 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eae9ab4e6294455780b1f8ead3623326. Latest consensus state: current_term: 1 leader_uuid: "eae9ab4e6294455780b1f8ead3623326" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae9ab4e6294455780b1f8ead3623326" member_type: VOTER } }
I20240502 07:59:29.299571 654 sys_catalog.cc:458] T 00000000000000000000000000000000 P eae9ab4e6294455780b1f8ead3623326 [sys.catalog]: This master's current role is: LEADER
I20240502 07:59:29.299443 653 sys_catalog.cc:455] T 00000000000000000000000000000000 P eae9ab4e6294455780b1f8ead3623326 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eae9ab4e6294455780b1f8ead3623326" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae9ab4e6294455780b1f8ead3623326" member_type: VOTER } }
I20240502 07:59:29.299930 661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 07:59:29.300184 653 sys_catalog.cc:458] T 00000000000000000000000000000000 P eae9ab4e6294455780b1f8ead3623326 [sys.catalog]: This master's current role is: LEADER
I20240502 07:59:29.301159 661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 07:59:29.301443 30122 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 07:59:29.303345 661 catalog_manager.cc:1347] Generated new cluster ID: 29fc80c8eb4e4d69b51a4b0e253e6f1a
I20240502 07:59:29.303408 661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 07:59:29.311807 661 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 07:59:29.312479 661 catalog_manager.cc:1504] Loading token signing keys...
I20240502 07:59:29.324441 661 catalog_manager.cc:5857] T 00000000000000000000000000000000 P eae9ab4e6294455780b1f8ead3623326: Generated new TSK 0
I20240502 07:59:29.324620 661 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 07:59:29.334273 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:29.337409 670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:29.338236 671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:29.338986 673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:29.339656 30122 server_base.cc:1034] running on GCE node
I20240502 07:59:29.339846 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:59:29.339890 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:59:29.339915 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636769339914 us; error 0 us; skew 500 ppm
I20240502 07:59:29.340971 30122 webserver.cc:460] Webserver started at http://127.29.106.129:42903/ using document root <none> and password file <none>
I20240502 07:59:29.341131 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:29.341187 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:29.341265 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:29.341675 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636688849691-30122-0/minicluster-data/ts-0-root/instance:
uuid: "4a53d69263294ab7bab0c59f782cd97b"
format_stamp: "Formatted at 2024-05-02 07:59:29 on dist-test-slave-m6l2"
I20240502 07:59:29.343509 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240502 07:59:29.344691 678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:29.345013 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:29.345113 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636688849691-30122-0/minicluster-data/ts-0-root
uuid: "4a53d69263294ab7bab0c59f782cd97b"
format_stamp: "Formatted at 2024-05-02 07:59:29 on dist-test-slave-m6l2"
I20240502 07:59:29.345214 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636688849691-30122-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636688849691-30122-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636688849691-30122-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:29.367197 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:29.367699 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:29.368031 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:29.368618 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:29.368660 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:29.368710 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:29.368743 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:29.373564 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.129:45827
I20240502 07:59:29.373667 741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.129:45827 every 8 connection(s)
I20240502 07:59:29.374532 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:29.377445 746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:29.378078 747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:29.378734 749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:29.379415 30122 server_base.cc:1034] running on GCE node
I20240502 07:59:29.379441 742 heartbeater.cc:347] Connected to a master server at 127.29.106.190:36227
I20240502 07:59:29.379577 742 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:29.379640 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:59:29.379709 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:59:29.379740 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636769379740 us; error 0 us; skew 500 ppm
I20240502 07:59:29.379913 742 heartbeater.cc:511] Master 127.29.106.190:36227 requested a full tablet report, sending...
I20240502 07:59:29.380766 613 ts_manager.cc:194] Registered new tserver with Master: 4a53d69263294ab7bab0c59f782cd97b (127.29.106.129:45827)
I20240502 07:59:29.381050 30122 webserver.cc:460] Webserver started at http://127.29.106.130:43291/ using document root <none> and password file <none>
I20240502 07:59:29.381203 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:29.381260 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:29.381350 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:29.381721 613 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40690
I20240502 07:59:29.381728 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636688849691-30122-0/minicluster-data/ts-1-root/instance:
uuid: "c14fa43abb7648c3b4f39d7c8b33b61f"
format_stamp: "Formatted at 2024-05-02 07:59:29 on dist-test-slave-m6l2"
I20240502 07:59:29.383258 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:29.384193 754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:29.384630 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:29.384716 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636688849691-30122-0/minicluster-data/ts-1-root
uuid: "c14fa43abb7648c3b4f39d7c8b33b61f"
format_stamp: "Formatted at 2024-05-02 07:59:29 on dist-test-slave-m6l2"
I20240502 07:59:29.384796 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636688849691-30122-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636688849691-30122-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636688849691-30122-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:29.415715 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:29.416198 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:29.416594 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:29.417116 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:29.417165 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:29.417199 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:29.417232 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:29.421720 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.130:44195
I20240502 07:59:29.421823 817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.130:44195 every 8 connection(s)
I20240502 07:59:29.422566 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:29.425433 821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:29.425983 822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:29.426637 824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:29.427227 30122 server_base.cc:1034] running on GCE node
I20240502 07:59:29.427421 818 heartbeater.cc:347] Connected to a master server at 127.29.106.190:36227
I20240502 07:59:29.427496 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240502 07:59:29.427542 818 heartbeater.cc:464] Registering TS with master...
W20240502 07:59:29.427547 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:59:29.427603 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636769427603 us; error 0 us; skew 500 ppm
I20240502 07:59:29.427781 818 heartbeater.cc:511] Master 127.29.106.190:36227 requested a full tablet report, sending...
I20240502 07:59:29.428481 613 ts_manager.cc:194] Registered new tserver with Master: c14fa43abb7648c3b4f39d7c8b33b61f (127.29.106.130:44195)
I20240502 07:59:29.428779 30122 webserver.cc:460] Webserver started at http://127.29.106.131:37941/ using document root <none> and password file <none>
I20240502 07:59:29.428946 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:29.428999 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:29.429083 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:29.429116 613 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40700
I20240502 07:59:29.429682 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636688849691-30122-0/minicluster-data/ts-2-root/instance:
uuid: "e7dccef5728c40219a7cad7971509690"
format_stamp: "Formatted at 2024-05-02 07:59:29 on dist-test-slave-m6l2"
I20240502 07:59:29.431267 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 07:59:29.432250 829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:29.432606 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:29.432687 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636688849691-30122-0/minicluster-data/ts-2-root
uuid: "e7dccef5728c40219a7cad7971509690"
format_stamp: "Formatted at 2024-05-02 07:59:29 on dist-test-slave-m6l2"
I20240502 07:59:29.432762 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636688849691-30122-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636688849691-30122-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636688849691-30122-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:29.445325 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:29.445761 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:29.446085 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:29.446652 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:29.446697 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:29.446748 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:29.446781 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:29.452159 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.131:34105
I20240502 07:59:29.452257 892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.131:34105 every 8 connection(s)
I20240502 07:59:29.457211 893 heartbeater.cc:347] Connected to a master server at 127.29.106.190:36227
I20240502 07:59:29.457351 893 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:29.457687 893 heartbeater.cc:511] Master 127.29.106.190:36227 requested a full tablet report, sending...
I20240502 07:59:29.458300 613 ts_manager.cc:194] Registered new tserver with Master: e7dccef5728c40219a7cad7971509690 (127.29.106.131:34105)
I20240502 07:59:29.458431 30122 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005737345s
I20240502 07:59:29.458848 613 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40710
I20240502 07:59:30.382759 742 heartbeater.cc:503] Master 127.29.106.190:36227 was elected leader, sending a full tablet report...
I20240502 07:59:30.430214 818 heartbeater.cc:503] Master 127.29.106.190:36227 was elected leader, sending a full tablet report...
I20240502 07:59:30.459829 893 heartbeater.cc:503] Master 127.29.106.190:36227 was elected leader, sending a full tablet report...
I20240502 07:59:30.491310 30122 test_util.cc:268] Using random seed: -73417814
I20240502 07:59:30.501268 613 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40726:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 07:59:30.501797 613 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 07:59:30.513006 782 tablet_service.cc:1467] Processing CreateTablet for tablet 8f394de0c470455f9fa955d4e58df036 (DEFAULT_TABLE table=test-workload [id=217634fcfa6e44a8a74cfabd09b27287]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:30.513362 782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f394de0c470455f9fa955d4e58df036. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:30.513734 706 tablet_service.cc:1467] Processing CreateTablet for tablet 8f394de0c470455f9fa955d4e58df036 (DEFAULT_TABLE table=test-workload [id=217634fcfa6e44a8a74cfabd09b27287]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:30.514098 706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f394de0c470455f9fa955d4e58df036. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:30.515904 914 tablet_bootstrap.cc:492] T 8f394de0c470455f9fa955d4e58df036 P c14fa43abb7648c3b4f39d7c8b33b61f: Bootstrap starting.
I20240502 07:59:30.516563 915 tablet_bootstrap.cc:492] T 8f394de0c470455f9fa955d4e58df036 P 4a53d69263294ab7bab0c59f782cd97b: Bootstrap starting.
I20240502 07:59:30.516644 857 tablet_service.cc:1467] Processing CreateTablet for tablet 8f394de0c470455f9fa955d4e58df036 (DEFAULT_TABLE table=test-workload [id=217634fcfa6e44a8a74cfabd09b27287]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:30.516920 857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f394de0c470455f9fa955d4e58df036. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:30.517194 914 tablet_bootstrap.cc:654] T 8f394de0c470455f9fa955d4e58df036 P c14fa43abb7648c3b4f39d7c8b33b61f: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:30.517663 915 tablet_bootstrap.cc:654] T 8f394de0c470455f9fa955d4e58df036 P 4a53d69263294ab7bab0c59f782cd97b: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:30.518602 914 tablet_bootstrap.cc:492] T 8f394de0c470455f9fa955d4e58df036 P c14fa43abb7648c3b4f39d7c8b33b61f: No bootstrap required, opened a new log
I20240502 07:59:30.518707 914 ts_tablet_manager.cc:1367] T 8f394de0c470455f9fa955d4e58df036 P c14fa43abb7648c3b4f39d7c8b33b61f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:30.518957 915 tablet_bootstrap.cc:492] T 8f394de0c470455f9fa955d4e58df036 P 4a53d69263294ab7bab0c59f782cd97b: No bootstrap required, opened a new log
I20240502 07:59:30.519057 915 ts_tablet_manager.cc:1367] T 8f394de0c470455f9fa955d4e58df036 P 4a53d69263294ab7bab0c59f782cd97b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:30.519203 914 raft_consensus.cc:348] T 8f394de0c470455f9fa955d4e58df036 P c14fa43abb7648c3b4f39d7c8b33b61f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c14fa43abb7648c3b4f39d7c8b33b61f" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 44195 } } peers { permanent_uuid: "e7dccef5728c40219a7cad7971509690" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34105 } } peers { permanent_uuid: "4a53d69263294ab7bab0c59f782cd97b" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 45827 } }
I20240502 07:59:30.519374 914 raft_consensus.cc:374] T 8f394de0c470455f9fa955d4e58df036 P c14fa43abb7648c3b4f39d7c8b33b61f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:30.519415 914 raft_consensus.cc:729] T 8f394de0c470455f9fa955d4e58df036 P c14fa43abb7648c3b4f39d7c8b33b61f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c14fa43abb7648c3b4f39d7c8b33b61f, State: Initialized, Role: FOLLOWER
I20240502 07:59:30.519544 915 raft_consensus.cc:348] T 8f394de0c470455f9fa955d4e58df036 P 4a53d69263294ab7bab0c59f782cd97b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c14fa43abb7648c3b4f39d7c8b33b61f" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 44195 } } peers { permanent_uuid: "e7dccef5728c40219a7cad7971509690" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34105 } } peers { permanent_uuid: "4a53d69263294ab7bab0c59f782cd97b" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 45827 } }
I20240502 07:59:30.519558 914 consensus_queue.cc:260] T 8f394de0c470455f9fa955d4e58df036 P c14fa43abb7648c3b4f39d7c8b33b61f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c14fa43abb7648c3b4f39d7c8b33b61f" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 44195 } } peers { permanent_uuid: "e7dccef5728c40219a7cad7971509690" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34105 } } peers { permanent_uuid: "4a53d69263294ab7bab0c59f782cd97b" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 45827 } }
I20240502 07:59:30.519699 915 raft_consensus.cc:374] T 8f394de0c470455f9fa955d4e58df036 P 4a53d69263294ab7bab0c59f782cd97b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:30.519765 915 raft_consensus.cc:729] T 8f394de0c470455f9fa955d4e58df036 P 4a53d69263294ab7bab0c59f782cd97b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a53d69263294ab7bab0c59f782cd97b, State: Initialized, Role: FOLLOWER
I20240502 07:59:30.519819 918 tablet_bootstrap.cc:492] T 8f394de0c470455f9fa955d4e58df036 P e7dccef5728c40219a7cad7971509690: Bootstrap starting.
I20240502 07:59:30.519919 915 consensus_queue.cc:260] T 8f394de0c470455f9fa955d4e58df036 P 4a53d69263294ab7bab0c59f782cd97b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c14fa43abb7648c3b4f39d7c8b33b61f" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 44195 } } peers { permanent_uuid: "e7dccef5728c40219a7cad7971509690" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34105 } } peers { permanent_uuid: "4a53d69263294ab7bab0c59f782cd97b" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 45827 } }
I20240502 07:59:30.520022 914 ts_tablet_manager.cc:1398] T 8f394de0c470455f9fa955d4e58df036 P c14fa43abb7648c3b4f39d7c8b33b61f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:59:30.520395 915 ts_tablet_manager.cc:1398] T 8f394de0c470455f9fa955d4e58df036 P 4a53d69263294ab7bab0c59f782cd97b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:59:30.521441 918 tablet_bootstrap.cc:654] T 8f394de0c470455f9fa955d4e58df036 P e7dccef5728c40219a7cad7971509690: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:30.523447 918 tablet_bootstrap.cc:492] T 8f394de0c470455f9fa955d4e58df036 P e7dccef5728c40219a7cad7971509690: No bootstrap required, opened a new log
I20240502 07:59:30.523530 918 ts_tablet_manager.cc:1367] T 8f394de0c470455f9fa955d4e58df036 P e7dccef5728c40219a7cad7971509690: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:59:30.523913 918 raft_consensus.cc:348] T 8f394de0c470455f9fa955d4e58df036 P e7dccef5728c40219a7cad7971509690 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c14fa43abb7648c3b4f39d7c8b33b61f" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 44195 } } peers { permanent_uuid: "e7dccef5728c40219a7cad7971509690" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34105 } } peers { permanent_uuid: "4a53d69263294ab7bab0c59f782cd97b" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 45827 } }
I20240502 07:59:30.524003 918 raft_consensus.cc:374] T 8f394de0c470455f9fa955d4e58df036 P e7dccef5728c40219a7cad7971509690 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:30.524027 918 raft_consensus.cc:729] T 8f394de0c470455f9fa955d4e58df036 P e7dccef5728c40219a7cad7971509690 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7dccef5728c40219a7cad7971509690, State: Initialized, Role: FOLLOWER
I20240502 07:59:30.524168 918 consensus_queue.cc:260] T 8f394de0c470455f9fa955d4e58df036 P e7dccef5728c40219a7cad7971509690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c14fa43abb7648c3b4f39d7c8b33b61f" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 44195 } } peers { permanent_uuid: "e7dccef5728c40219a7cad7971509690" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34105 } } peers { permanent_uuid: "4a53d69263294ab7bab0c59f782cd97b" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 45827 } }
I20240502 07:59:30.524544 918 ts_tablet_manager.cc:1398] T 8f394de0c470455f9fa955d4e58df036 P e7dccef5728c40219a7cad7971509690: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:30.600253 919 raft_consensus.cc:482] T 8f394de0c470455f9fa955d4e58df036 P c14fa43abb7648c3b4f39d7c8b33b61f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:30.600525 919 raft_consensus.cc:504] T 8f394de0c470455f9fa955d4e58df036 P c14fa43abb7648c3b4f39d7c8b33b61f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c14fa43abb7648c3b4f39d7c8b33b61f" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 44195 } } peers { permanent_uuid: "e7dccef5728c40219a7cad7971509690" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34105 } } peers { permanent_uuid: "4a53d69263294ab7bab0c59f782cd97b" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 45827 } }
I20240502 07:59:30.601142 919 leader_election.cc:288] T 8f394de0c470455f9fa955d4e58df036 P c14fa43abb7648c3b4f39d7c8b33b61f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7dccef5728c40219a7cad7971509690 (127.29.106.131:34105), 4a53d69263294ab7bab0c59f782cd97b (127.29.106.129:45827)
I20240502 07:59:30.606489 716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f394de0c470455f9fa955d4e58df036" candidate_uuid: "c14fa43abb7648c3b4f39d7c8b33b61f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a53d69263294ab7bab0c59f782cd97b" is_pre_election: true
I20240502 07:59:30.606547 867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f394de0c470455f9fa955d4e58df036" candidate_uuid: "c14fa43abb7648c3b4f39d7c8b33b61f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7dccef5728c40219a7cad7971509690" is_pre_election: true
I20240502 07:59:30.606678 716 raft_consensus.cc:2454] T 8f394de0c470455f9fa955d4e58df036 P 4a53d69263294ab7bab0c59f782cd97b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c14fa43abb7648c3b4f39d7c8b33b61f in term 0.
I20240502 07:59:30.606768 867 raft_consensus.cc:2454] T 8f394de0c470455f9fa955d4e58df036 P e7dccef5728c40219a7cad7971509690 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c14fa43abb7648c3b4f39d7c8b33b61f in term 0.
I20240502 07:59:30.607030 756 leader_election.cc:302] T 8f394de0c470455f9fa955d4e58df036 P c14fa43abb7648c3b4f39d7c8b33b61f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a53d69263294ab7bab0c59f782cd97b, c14fa43abb7648c3b4f39d7c8b33b61f; no voters:
I20240502 07:59:30.607244 919 raft_consensus.cc:2781] T 8f394de0c470455f9fa955d4e58df036 P c14fa43abb7648c3b4f39d7c8b33b61f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:30.607321 919 raft_consensus.cc:482] T 8f394de0c470455f9fa955d4e58df036 P c14fa43abb7648c3b4f39d7c8b33b61f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:30.607362 919 raft_consensus.cc:3037] T 8f394de0c470455f9fa955d4e58df036 P c14fa43abb7648c3b4f39d7c8b33b61f [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:30.608234 919 raft_consensus.cc:504] T 8f394de0c470455f9fa955d4e58df036 P c14fa43abb7648c3b4f39d7c8b33b61f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c14fa43abb7648c3b4f39d7c8b33b61f" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 44195 } } peers { permanent_uuid: "e7dccef5728c40219a7cad7971509690" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34105 } } peers { permanent_uuid: "4a53d69263294ab7bab0c59f782cd97b" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 45827 } }
I20240502 07:59:30.608578 919 leader_election.cc:288] T 8f394de0c470455f9fa955d4e58df036 P c14fa43abb7648c3b4f39d7c8b33b61f [CANDIDATE]: Term 1 election: Requested vote from peers e7dccef5728c40219a7cad7971509690 (127.29.106.131:34105), 4a53d69263294ab7bab0c59f782cd97b (127.29.106.129:45827)
I20240502 07:59:30.608898 867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f394de0c470455f9fa955d4e58df036" candidate_uuid: "c14fa43abb7648c3b4f39d7c8b33b61f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7dccef5728c40219a7cad7971509690"
I20240502 07:59:30.608930 716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f394de0c470455f9fa955d4e58df036" candidate_uuid: "c14fa43abb7648c3b4f39d7c8b33b61f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a53d69263294ab7bab0c59f782cd97b"
I20240502 07:59:30.609030 867 raft_consensus.cc:3037] T 8f394de0c470455f9fa955d4e58df036 P e7dccef5728c40219a7cad7971509690 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:30.609030 716 raft_consensus.cc:3037] T 8f394de0c470455f9fa955d4e58df036 P 4a53d69263294ab7bab0c59f782cd97b [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:30.610018 716 raft_consensus.cc:2454] T 8f394de0c470455f9fa955d4e58df036 P 4a53d69263294ab7bab0c59f782cd97b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c14fa43abb7648c3b4f39d7c8b33b61f in term 1.
I20240502 07:59:30.610018 867 raft_consensus.cc:2454] T 8f394de0c470455f9fa955d4e58df036 P e7dccef5728c40219a7cad7971509690 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c14fa43abb7648c3b4f39d7c8b33b61f in term 1.
I20240502 07:59:30.610378 758 leader_election.cc:302] T 8f394de0c470455f9fa955d4e58df036 P c14fa43abb7648c3b4f39d7c8b33b61f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c14fa43abb7648c3b4f39d7c8b33b61f, e7dccef5728c40219a7cad7971509690; no voters:
I20240502 07:59:30.610543 919 raft_consensus.cc:2781] T 8f394de0c470455f9fa955d4e58df036 P c14fa43abb7648c3b4f39d7c8b33b61f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:30.610757 919 raft_consensus.cc:686] T 8f394de0c470455f9fa955d4e58df036 P c14fa43abb7648c3b4f39d7c8b33b61f [term 1 LEADER]: Becoming Leader. State: Replica: c14fa43abb7648c3b4f39d7c8b33b61f, State: Running, Role: LEADER
I20240502 07:59:30.610893 919 consensus_queue.cc:237] T 8f394de0c470455f9fa955d4e58df036 P c14fa43abb7648c3b4f39d7c8b33b61f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c14fa43abb7648c3b4f39d7c8b33b61f" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 44195 } } peers { permanent_uuid: "e7dccef5728c40219a7cad7971509690" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34105 } } peers { permanent_uuid: "4a53d69263294ab7bab0c59f782cd97b" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 45827 } }
I20240502 07:59:30.612540 611 catalog_manager.cc:5484] T 8f394de0c470455f9fa955d4e58df036 P c14fa43abb7648c3b4f39d7c8b33b61f reported cstate change: term changed from 0 to 1, leader changed from <none> to c14fa43abb7648c3b4f39d7c8b33b61f (127.29.106.130). New cstate: current_term: 1 leader_uuid: "c14fa43abb7648c3b4f39d7c8b33b61f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c14fa43abb7648c3b4f39d7c8b33b61f" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 44195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7dccef5728c40219a7cad7971509690" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a53d69263294ab7bab0c59f782cd97b" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 45827 } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:31.031842 919 consensus_queue.cc:1035] T 8f394de0c470455f9fa955d4e58df036 P c14fa43abb7648c3b4f39d7c8b33b61f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a53d69263294ab7bab0c59f782cd97b" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:31.035533 930 consensus_queue.cc:1035] T 8f394de0c470455f9fa955d4e58df036 P c14fa43abb7648c3b4f39d7c8b33b61f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7dccef5728c40219a7cad7971509690" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 34105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:31.300980 667 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:59:31.301321 667 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 07:59:31.301409 667 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:59:31.301577 660 placement_policy_util.cc:407] tablet 8f394de0c470455f9fa955d4e58df036: detected majority of replicas (2 of 3) at location L0
I20240502 07:59:32.303737 660 placement_policy_util.cc:407] tablet 8f394de0c470455f9fa955d4e58df036: detected majority of replicas (2 of 3) at location L0
I20240502 07:59:33.301821 667 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:59:33.302217 667 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 07:59:33.302273 667 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:59:33.305420 660 placement_policy_util.cc:407] tablet 8f394de0c470455f9fa955d4e58df036: detected majority of replicas (2 of 3) at location L0
I20240502 07:59:34.307240 660 placement_policy_util.cc:407] tablet 8f394de0c470455f9fa955d4e58df036: detected majority of replicas (2 of 3) at location L0
I20240502 07:59:35.302652 667 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:59:35.303036 667 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 07:59:35.303093 667 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:59:35.308969 660 placement_policy_util.cc:407] tablet 8f394de0c470455f9fa955d4e58df036: detected majority of replicas (2 of 3) at location L0
I20240502 07:59:36.310803 660 placement_policy_util.cc:407] tablet 8f394de0c470455f9fa955d4e58df036: detected majority of replicas (2 of 3) at location L0
I20240502 07:59:37.303465 667 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:59:37.303856 667 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 07:59:37.303915 667 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:59:37.312638 660 placement_policy_util.cc:407] tablet 8f394de0c470455f9fa955d4e58df036: detected majority of replicas (2 of 3) at location L0
I20240502 07:59:38.314446 660 placement_policy_util.cc:407] tablet 8f394de0c470455f9fa955d4e58df036: detected majority of replicas (2 of 3) at location L0
I20240502 07:59:39.304348 667 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:59:39.304665 667 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 07:59:39.304937 667 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:59:39.316136 660 placement_policy_util.cc:407] tablet 8f394de0c470455f9fa955d4e58df036: detected majority of replicas (2 of 3) at location L0
I20240502 07:59:40.317909 660 placement_policy_util.cc:407] tablet 8f394de0c470455f9fa955d4e58df036: detected majority of replicas (2 of 3) at location L0
I20240502 07:59:41.305373 667 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:59:41.305675 667 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 07:59:41.305725 667 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:59:41.319738 660 placement_policy_util.cc:407] tablet 8f394de0c470455f9fa955d4e58df036: detected majority of replicas (2 of 3) at location L0
I20240502 07:59:41.684689 30122 tablet_server.cc:178] TabletServer@127.29.106.129:0 shutting down...
I20240502 07:59:41.688333 30122 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 07:59:41.688575 30122 tablet_replica.cc:330] T 8f394de0c470455f9fa955d4e58df036 P 4a53d69263294ab7bab0c59f782cd97b: stopping tablet replica
I20240502 07:59:41.688699 30122 raft_consensus.cc:2229] T 8f394de0c470455f9fa955d4e58df036 P 4a53d69263294ab7bab0c59f782cd97b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:41.688818 30122 raft_consensus.cc:2258] T 8f394de0c470455f9fa955d4e58df036 P 4a53d69263294ab7bab0c59f782cd97b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:41.701689 30122 tablet_server.cc:194] TabletServer@127.29.106.129:0 shutdown complete.
I20240502 07:59:41.703917 30122 tablet_server.cc:178] TabletServer@127.29.106.130:0 shutting down...
I20240502 07:59:41.707554 30122 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 07:59:41.707793 30122 tablet_replica.cc:330] T 8f394de0c470455f9fa955d4e58df036 P c14fa43abb7648c3b4f39d7c8b33b61f: stopping tablet replica
I20240502 07:59:41.707926 30122 raft_consensus.cc:2229] T 8f394de0c470455f9fa955d4e58df036 P c14fa43abb7648c3b4f39d7c8b33b61f [term 1 LEADER]: Raft consensus shutting down.
I20240502 07:59:41.708115 30122 raft_consensus.cc:2258] T 8f394de0c470455f9fa955d4e58df036 P c14fa43abb7648c3b4f39d7c8b33b61f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:41.720904 30122 tablet_server.cc:194] TabletServer@127.29.106.130:0 shutdown complete.
I20240502 07:59:41.722956 30122 tablet_server.cc:178] TabletServer@127.29.106.131:0 shutting down...
I20240502 07:59:41.727576 30122 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 07:59:41.727798 30122 tablet_replica.cc:330] T 8f394de0c470455f9fa955d4e58df036 P e7dccef5728c40219a7cad7971509690: stopping tablet replica
I20240502 07:59:41.727922 30122 raft_consensus.cc:2229] T 8f394de0c470455f9fa955d4e58df036 P e7dccef5728c40219a7cad7971509690 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:41.728053 30122 raft_consensus.cc:2258] T 8f394de0c470455f9fa955d4e58df036 P e7dccef5728c40219a7cad7971509690 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:41.740816 30122 tablet_server.cc:194] TabletServer@127.29.106.131:0 shutdown complete.
I20240502 07:59:41.742897 30122 master.cc:537] Master@127.29.106.190:36227 shutting down...
I20240502 07:59:41.746505 30122 raft_consensus.cc:2229] T 00000000000000000000000000000000 P eae9ab4e6294455780b1f8ead3623326 [term 1 LEADER]: Raft consensus shutting down.
I20240502 07:59:41.746683 30122 raft_consensus.cc:2258] T 00000000000000000000000000000000 P eae9ab4e6294455780b1f8ead3623326 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:41.746753 30122 tablet_replica.cc:330] T 00000000000000000000000000000000 P eae9ab4e6294455780b1f8ead3623326: stopping tablet replica
I20240502 07:59:41.759405 30122 master.cc:552] Master@127.29.106.190:36227 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (12526 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
I20240502 07:59:41.766279 30122 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.106.190:35263
I20240502 07:59:41.766636 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:41.769170 942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:41.769830 940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:41.770428 939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:41.771116 30122 server_base.cc:1034] running on GCE node
I20240502 07:59:41.771278 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:59:41.771323 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:59:41.771342 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636781771341 us; error 0 us; skew 500 ppm
I20240502 07:59:41.776783 30122 webserver.cc:460] Webserver started at http://127.29.106.190:38281/ using document root <none> and password file <none>
I20240502 07:59:41.776958 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:41.777017 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:41.777087 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:41.777592 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/master-0-root/instance:
uuid: "004b7395c3984a0d9d1918160d8b4233"
format_stamp: "Formatted at 2024-05-02 07:59:41 on dist-test-slave-m6l2"
I20240502 07:59:41.779181 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240502 07:59:41.780208 947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:41.780498 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:41.780560 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/master-0-root
uuid: "004b7395c3984a0d9d1918160d8b4233"
format_stamp: "Formatted at 2024-05-02 07:59:41 on dist-test-slave-m6l2"
I20240502 07:59:41.780616 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:41.788097 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:41.788478 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:41.792960 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.190:35263
I20240502 07:59:41.795121 999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.190:35263 every 8 connection(s)
I20240502 07:59:41.795660 1000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:41.797508 1000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 004b7395c3984a0d9d1918160d8b4233: Bootstrap starting.
I20240502 07:59:41.798260 1000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 004b7395c3984a0d9d1918160d8b4233: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:41.799330 1000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 004b7395c3984a0d9d1918160d8b4233: No bootstrap required, opened a new log
I20240502 07:59:41.799700 1000 raft_consensus.cc:348] T 00000000000000000000000000000000 P 004b7395c3984a0d9d1918160d8b4233 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004b7395c3984a0d9d1918160d8b4233" member_type: VOTER }
I20240502 07:59:41.799799 1000 raft_consensus.cc:374] T 00000000000000000000000000000000 P 004b7395c3984a0d9d1918160d8b4233 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:41.799822 1000 raft_consensus.cc:729] T 00000000000000000000000000000000 P 004b7395c3984a0d9d1918160d8b4233 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 004b7395c3984a0d9d1918160d8b4233, State: Initialized, Role: FOLLOWER
I20240502 07:59:41.799934 1000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 004b7395c3984a0d9d1918160d8b4233 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004b7395c3984a0d9d1918160d8b4233" member_type: VOTER }
I20240502 07:59:41.800020 1000 raft_consensus.cc:388] T 00000000000000000000000000000000 P 004b7395c3984a0d9d1918160d8b4233 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:59:41.800045 1000 raft_consensus.cc:482] T 00000000000000000000000000000000 P 004b7395c3984a0d9d1918160d8b4233 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:59:41.800081 1000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 004b7395c3984a0d9d1918160d8b4233 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:41.800912 1000 raft_consensus.cc:504] T 00000000000000000000000000000000 P 004b7395c3984a0d9d1918160d8b4233 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004b7395c3984a0d9d1918160d8b4233" member_type: VOTER }
I20240502 07:59:41.801119 1000 leader_election.cc:302] T 00000000000000000000000000000000 P 004b7395c3984a0d9d1918160d8b4233 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 004b7395c3984a0d9d1918160d8b4233; no voters:
I20240502 07:59:41.801303 1000 leader_election.cc:288] T 00000000000000000000000000000000 P 004b7395c3984a0d9d1918160d8b4233 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:59:41.801429 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 004b7395c3984a0d9d1918160d8b4233 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:41.801623 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 004b7395c3984a0d9d1918160d8b4233 [term 1 LEADER]: Becoming Leader. State: Replica: 004b7395c3984a0d9d1918160d8b4233, State: Running, Role: LEADER
I20240502 07:59:41.801774 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 004b7395c3984a0d9d1918160d8b4233 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004b7395c3984a0d9d1918160d8b4233" member_type: VOTER }
I20240502 07:59:41.801805 1000 sys_catalog.cc:564] T 00000000000000000000000000000000 P 004b7395c3984a0d9d1918160d8b4233 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 07:59:41.802314 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 004b7395c3984a0d9d1918160d8b4233 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "004b7395c3984a0d9d1918160d8b4233" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004b7395c3984a0d9d1918160d8b4233" member_type: VOTER } }
I20240502 07:59:41.802402 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 004b7395c3984a0d9d1918160d8b4233 [sys.catalog]: This master's current role is: LEADER
I20240502 07:59:41.802409 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 004b7395c3984a0d9d1918160d8b4233 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 004b7395c3984a0d9d1918160d8b4233. Latest consensus state: current_term: 1 leader_uuid: "004b7395c3984a0d9d1918160d8b4233" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004b7395c3984a0d9d1918160d8b4233" member_type: VOTER } }
I20240502 07:59:41.802732 1007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 07:59:41.802788 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 004b7395c3984a0d9d1918160d8b4233 [sys.catalog]: This master's current role is: LEADER
I20240502 07:59:41.803726 1007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 07:59:41.804554 30122 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 07:59:41.806128 1007 catalog_manager.cc:1347] Generated new cluster ID: 9ae4457ec90d4e64ac82c11a552acdb8
I20240502 07:59:41.806187 1007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 07:59:41.841181 1007 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 07:59:41.841842 1007 catalog_manager.cc:1504] Loading token signing keys...
I20240502 07:59:41.853799 1007 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 004b7395c3984a0d9d1918160d8b4233: Generated new TSK 0
I20240502 07:59:41.854014 1007 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 07:59:41.869735 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:41.872596 1022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:41.873250 1021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:41.873872 1024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:41.874616 30122 server_base.cc:1034] running on GCE node
I20240502 07:59:41.874801 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:59:41.874842 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:59:41.874858 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636781874858 us; error 0 us; skew 500 ppm
I20240502 07:59:41.875809 30122 webserver.cc:460] Webserver started at http://127.29.106.129:39741/ using document root <none> and password file <none>
I20240502 07:59:41.875948 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:41.875993 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:41.876056 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:41.876464 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-0-root/instance:
uuid: "77e648af9dbb46fc881340efe1d1e3c3"
format_stamp: "Formatted at 2024-05-02 07:59:41 on dist-test-slave-m6l2"
I20240502 07:59:41.877867 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:41.878844 1029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:41.879122 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:41.879190 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-0-root
uuid: "77e648af9dbb46fc881340efe1d1e3c3"
format_stamp: "Formatted at 2024-05-02 07:59:41 on dist-test-slave-m6l2"
I20240502 07:59:41.879261 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:41.899145 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:41.899533 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:41.899830 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:41.900327 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:41.900369 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:41.900501 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:41.900571 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:41.905333 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.129:34761
I20240502 07:59:41.905424 1092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.129:34761 every 8 connection(s)
I20240502 07:59:41.906370 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:41.909291 1097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:41.909981 1098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:41.910698 1100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:41.910897 1093 heartbeater.cc:347] Connected to a master server at 127.29.106.190:35263
I20240502 07:59:41.910984 1093 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:41.911276 1093 heartbeater.cc:511] Master 127.29.106.190:35263 requested a full tablet report, sending...
I20240502 07:59:41.911299 30122 server_base.cc:1034] running on GCE node
I20240502 07:59:41.911463 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:59:41.911506 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:59:41.911530 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636781911530 us; error 0 us; skew 500 ppm
I20240502 07:59:41.911988 964 ts_manager.cc:194] Registered new tserver with Master: 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761)
I20240502 07:59:41.912637 30122 webserver.cc:460] Webserver started at http://127.29.106.130:40627/ using document root <none> and password file <none>
I20240502 07:59:41.912792 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:41.912845 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:41.912912 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:41.913051 964 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40004
I20240502 07:59:41.913352 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-1-root/instance:
uuid: "b1e9945160714b16b2db91f90c5ccc70"
format_stamp: "Formatted at 2024-05-02 07:59:41 on dist-test-slave-m6l2"
I20240502 07:59:41.914897 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 07:59:41.915904 1105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:41.916148 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:41.916218 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-1-root
uuid: "b1e9945160714b16b2db91f90c5ccc70"
format_stamp: "Formatted at 2024-05-02 07:59:41 on dist-test-slave-m6l2"
I20240502 07:59:41.916323 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:41.927791 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:41.928174 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:41.928532 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:41.929235 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:41.929287 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:41.929579 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:41.929617 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:41.933825 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.130:38203
I20240502 07:59:41.933846 1168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.130:38203 every 8 connection(s)
I20240502 07:59:41.934574 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:41.937268 1172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:41.937904 1173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:41.938624 1175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:41.938722 1169 heartbeater.cc:347] Connected to a master server at 127.29.106.190:35263
I20240502 07:59:41.938817 1169 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:41.939038 1169 heartbeater.cc:511] Master 127.29.106.190:35263 requested a full tablet report, sending...
I20240502 07:59:41.939234 30122 server_base.cc:1034] running on GCE node
I20240502 07:59:41.939379 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:59:41.939424 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:59:41.939445 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636781939444 us; error 0 us; skew 500 ppm
I20240502 07:59:41.939703 964 ts_manager.cc:194] Registered new tserver with Master: b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203)
I20240502 07:59:41.940395 964 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40020
I20240502 07:59:41.940505 30122 webserver.cc:460] Webserver started at http://127.29.106.131:42617/ using document root <none> and password file <none>
I20240502 07:59:41.940657 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:41.940730 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:41.940809 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:41.941254 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-2-root/instance:
uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
format_stamp: "Formatted at 2024-05-02 07:59:41 on dist-test-slave-m6l2"
I20240502 07:59:41.942910 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 07:59:41.943804 1180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:41.944015 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:41.944090 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-2-root
uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
format_stamp: "Formatted at 2024-05-02 07:59:41 on dist-test-slave-m6l2"
I20240502 07:59:41.944159 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:41.955809 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:41.956377 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:41.956755 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:41.957212 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:41.957294 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:41.957369 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:41.957405 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:41.962108 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.131:36737
I20240502 07:59:41.962165 1243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.131:36737 every 8 connection(s)
I20240502 07:59:41.966989 1244 heartbeater.cc:347] Connected to a master server at 127.29.106.190:35263
I20240502 07:59:41.967121 1244 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:41.967357 1244 heartbeater.cc:511] Master 127.29.106.190:35263 requested a full tablet report, sending...
I20240502 07:59:41.967969 964 ts_manager.cc:194] Registered new tserver with Master: 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:41.968130 30122 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005620013s
I20240502 07:59:41.968664 964 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40022
I20240502 07:59:42.914041 1093 heartbeater.cc:503] Master 127.29.106.190:35263 was elected leader, sending a full tablet report...
I20240502 07:59:42.941469 1169 heartbeater.cc:503] Master 127.29.106.190:35263 was elected leader, sending a full tablet report...
I20240502 07:59:42.969564 1244 heartbeater.cc:503] Master 127.29.106.190:35263 was elected leader, sending a full tablet report...
I20240502 07:59:42.994648 30122 test_util.cc:268] Using random seed: -60914475
I20240502 07:59:43.004638 964 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40028:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu""\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 07:59:43.005430 964 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 07:59:43.021677 1131 tablet_service.cc:1467] Processing CreateTablet for tablet 2ecebe8dad3946bd956afacf695fabaa (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240502 07:59:43.022078 1131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ecebe8dad3946bd956afacf695fabaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.022442 1130 tablet_service.cc:1467] Processing CreateTablet for tablet 682e26b333104f82863b92c545cfdafe (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240502 07:59:43.022572 1057 tablet_service.cc:1467] Processing CreateTablet for tablet 2ecebe8dad3946bd956afacf695fabaa (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240502 07:59:43.022991 1057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ecebe8dad3946bd956afacf695fabaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.023227 1132 tablet_service.cc:1467] Processing CreateTablet for tablet 58c5f52a420c467684b2cb50972e7c99 (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240502 07:59:43.023810 1129 tablet_service.cc:1467] Processing CreateTablet for tablet 5437f94679a841e3980053e69d624d76 (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240502 07:59:43.024252 1128 tablet_service.cc:1467] Processing CreateTablet for tablet d0d982dd52d246348fb3984fd1c3a40f (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240502 07:59:43.024762 1127 tablet_service.cc:1467] Processing CreateTablet for tablet cdf0ff2967c040a589f915172d3f446b (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240502 07:59:43.025249 1126 tablet_service.cc:1467] Processing CreateTablet for tablet c7c1536618ae4e74aa79349d04201fb9 (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240502 07:59:43.025686 1125 tablet_service.cc:1467] Processing CreateTablet for tablet 69a9ebd0c390436b8ace08089f719358 (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240502 07:59:43.026026 1056 tablet_service.cc:1467] Processing CreateTablet for tablet 682e26b333104f82863b92c545cfdafe (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240502 07:59:43.026111 1124 tablet_service.cc:1467] Processing CreateTablet for tablet 2c1b6523d6724a02acc9ae19f3f0255e (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240502 07:59:43.026458 1056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 682e26b333104f82863b92c545cfdafe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.026624 1049 tablet_service.cc:1467] Processing CreateTablet for tablet 69a9ebd0c390436b8ace08089f719358 (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240502 07:59:43.026921 1049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69a9ebd0c390436b8ace08089f719358. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.028515 1055 tablet_service.cc:1467] Processing CreateTablet for tablet fda98cfe14f947969fb41e26c75bc384 (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240502 07:59:43.028744 1055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fda98cfe14f947969fb41e26c75bc384. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.029942 1049 tablet_service.cc:1467] Processing CreateTablet for tablet 6edf17550e3f4a0b945477edc6de6af1 (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240502 07:59:43.030241 1049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6edf17550e3f4a0b945477edc6de6af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.030711 1054 tablet_service.cc:1467] Processing CreateTablet for tablet 58c5f52a420c467684b2cb50972e7c99 (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240502 07:59:43.030921 1054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58c5f52a420c467684b2cb50972e7c99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.022992 1133 tablet_service.cc:1467] Processing CreateTablet for tablet fda98cfe14f947969fb41e26c75bc384 (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240502 07:59:43.032841 1053 tablet_service.cc:1467] Processing CreateTablet for tablet 5437f94679a841e3980053e69d624d76 (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240502 07:59:43.033143 1053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5437f94679a841e3980053e69d624d76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.035599 1052 tablet_service.cc:1467] Processing CreateTablet for tablet d0d982dd52d246348fb3984fd1c3a40f (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240502 07:59:43.035889 1052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0d982dd52d246348fb3984fd1c3a40f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.038283 1051 tablet_service.cc:1467] Processing CreateTablet for tablet cdf0ff2967c040a589f915172d3f446b (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240502 07:59:43.038895 1050 tablet_service.cc:1467] Processing CreateTablet for tablet c7c1536618ae4e74aa79349d04201fb9 (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240502 07:59:43.039343 1048 tablet_service.cc:1467] Processing CreateTablet for tablet 2c1b6523d6724a02acc9ae19f3f0255e (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240502 07:59:43.039981 1051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdf0ff2967c040a589f915172d3f446b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.022704 1130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 682e26b333104f82863b92c545cfdafe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.040277 1050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7c1536618ae4e74aa79349d04201fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.040357 1132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58c5f52a420c467684b2cb50972e7c99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.040491 1126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7c1536618ae4e74aa79349d04201fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.040720 1127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdf0ff2967c040a589f915172d3f446b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.040769 1048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c1b6523d6724a02acc9ae19f3f0255e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.043606 1133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fda98cfe14f947969fb41e26c75bc384. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.043818 1127 tablet_service.cc:1467] Processing CreateTablet for tablet 6edf17550e3f4a0b945477edc6de6af1 (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240502 07:59:43.044103 1127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6edf17550e3f4a0b945477edc6de6af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.044348 1128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0d982dd52d246348fb3984fd1c3a40f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.046576 1208 tablet_service.cc:1467] Processing CreateTablet for tablet 2ecebe8dad3946bd956afacf695fabaa (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240502 07:59:43.046653 1206 tablet_service.cc:1467] Processing CreateTablet for tablet fda98cfe14f947969fb41e26c75bc384 (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240502 07:59:43.046924 1208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ecebe8dad3946bd956afacf695fabaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.047067 1129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5437f94679a841e3980053e69d624d76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.047068 1206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fda98cfe14f947969fb41e26c75bc384. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.047528 1207 tablet_service.cc:1467] Processing CreateTablet for tablet 682e26b333104f82863b92c545cfdafe (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240502 07:59:43.047806 1207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 682e26b333104f82863b92c545cfdafe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.050539 1207 tablet_service.cc:1467] Processing CreateTablet for tablet 6edf17550e3f4a0b945477edc6de6af1 (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240502 07:59:43.050815 1207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6edf17550e3f4a0b945477edc6de6af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.051487 1050 tablet_service.cc:1467] Processing CreateTablet for tablet df4233b549b34b01b6f9662c7e7b3515 (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240502 07:59:43.051718 1125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69a9ebd0c390436b8ace08089f719358. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.051740 1050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df4233b549b34b01b6f9662c7e7b3515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.052484 1205 tablet_service.cc:1467] Processing CreateTablet for tablet 58c5f52a420c467684b2cb50972e7c99 (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240502 07:59:43.052848 1205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58c5f52a420c467684b2cb50972e7c99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.053673 1124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c1b6523d6724a02acc9ae19f3f0255e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.054843 1131 tablet_service.cc:1467] Processing CreateTablet for tablet df4233b549b34b01b6f9662c7e7b3515 (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240502 07:59:43.055096 1131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df4233b549b34b01b6f9662c7e7b3515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.055231 1204 tablet_service.cc:1467] Processing CreateTablet for tablet 5437f94679a841e3980053e69d624d76 (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240502 07:59:43.055491 1204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5437f94679a841e3980053e69d624d76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.056627 1203 tablet_service.cc:1467] Processing CreateTablet for tablet d0d982dd52d246348fb3984fd1c3a40f (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240502 07:59:43.056939 1203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0d982dd52d246348fb3984fd1c3a40f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.057571 1264 tablet_bootstrap.cc:492] T 2ecebe8dad3946bd956afacf695fabaa P 77e648af9dbb46fc881340efe1d1e3c3: Bootstrap starting.
I20240502 07:59:43.058398 1264 tablet_bootstrap.cc:654] T 2ecebe8dad3946bd956afacf695fabaa P 77e648af9dbb46fc881340efe1d1e3c3: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.058780 1199 tablet_service.cc:1467] Processing CreateTablet for tablet 2c1b6523d6724a02acc9ae19f3f0255e (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240502 07:59:43.059077 1199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c1b6523d6724a02acc9ae19f3f0255e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.059098 1266 tablet_bootstrap.cc:492] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f: Bootstrap starting.
I20240502 07:59:43.059718 1264 tablet_bootstrap.cc:492] T 2ecebe8dad3946bd956afacf695fabaa P 77e648af9dbb46fc881340efe1d1e3c3: No bootstrap required, opened a new log
I20240502 07:59:43.059808 1264 ts_tablet_manager.cc:1367] T 2ecebe8dad3946bd956afacf695fabaa P 77e648af9dbb46fc881340efe1d1e3c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:43.059906 1266 tablet_bootstrap.cc:654] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.060396 1264 raft_consensus.cc:348] T 2ecebe8dad3946bd956afacf695fabaa P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.060513 1264 raft_consensus.cc:374] T 2ecebe8dad3946bd956afacf695fabaa P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.060526 1265 tablet_bootstrap.cc:492] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70: Bootstrap starting.
I20240502 07:59:43.060545 1264 raft_consensus.cc:729] T 2ecebe8dad3946bd956afacf695fabaa P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e648af9dbb46fc881340efe1d1e3c3, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.060649 1202 tablet_service.cc:1467] Processing CreateTablet for tablet cdf0ff2967c040a589f915172d3f446b (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240502 07:59:43.060722 1264 consensus_queue.cc:260] T 2ecebe8dad3946bd956afacf695fabaa P 77e648af9dbb46fc881340efe1d1e3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.060911 1202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdf0ff2967c040a589f915172d3f446b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.061148 1264 ts_tablet_manager.cc:1398] T 2ecebe8dad3946bd956afacf695fabaa P 77e648af9dbb46fc881340efe1d1e3c3: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:59:43.061424 1264 tablet_bootstrap.cc:492] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3: Bootstrap starting.
I20240502 07:59:43.061457 1265 tablet_bootstrap.cc:654] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.061656 1200 tablet_service.cc:1467] Processing CreateTablet for tablet 69a9ebd0c390436b8ace08089f719358 (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240502 07:59:43.061898 1200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69a9ebd0c390436b8ace08089f719358. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.062556 1265 tablet_bootstrap.cc:492] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70: No bootstrap required, opened a new log
I20240502 07:59:43.062587 1264 tablet_bootstrap.cc:654] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.062645 1265 ts_tablet_manager.cc:1367] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:43.063117 1265 raft_consensus.cc:348] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.063225 1265 raft_consensus.cc:374] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.063256 1265 raft_consensus.cc:729] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e9945160714b16b2db91f90c5ccc70, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.063411 1265 consensus_queue.cc:260] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.063918 1265 ts_tablet_manager.cc:1398] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:59:43.064116 1265 tablet_bootstrap.cc:492] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70: Bootstrap starting.
I20240502 07:59:43.064532 1201 tablet_service.cc:1467] Processing CreateTablet for tablet c7c1536618ae4e74aa79349d04201fb9 (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240502 07:59:43.064806 1201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7c1536618ae4e74aa79349d04201fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.050815 1206 tablet_service.cc:1467] Processing CreateTablet for tablet df4233b549b34b01b6f9662c7e7b3515 (DEFAULT_TABLE table=test-workload [id=c4044eda63f741838cbc708a8054d3b9]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240502 07:59:43.065452 1206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df4233b549b34b01b6f9662c7e7b3515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.065183 1265 tablet_bootstrap.cc:654] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.067724 1265 tablet_bootstrap.cc:492] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70: No bootstrap required, opened a new log
I20240502 07:59:43.067842 1265 ts_tablet_manager.cc:1367] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:59:43.068316 1265 raft_consensus.cc:348] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.068459 1265 raft_consensus.cc:374] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.068506 1265 raft_consensus.cc:729] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e9945160714b16b2db91f90c5ccc70, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.068643 1265 consensus_queue.cc:260] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.068950 1265 ts_tablet_manager.cc:1398] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:43.068959 1264 tablet_bootstrap.cc:492] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3: No bootstrap required, opened a new log
I20240502 07:59:43.069083 1264 ts_tablet_manager.cc:1367] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 07:59:43.069183 1265 tablet_bootstrap.cc:492] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70: Bootstrap starting.
I20240502 07:59:43.069516 1264 raft_consensus.cc:348] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.069897 1264 raft_consensus.cc:374] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.069940 1264 raft_consensus.cc:729] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e648af9dbb46fc881340efe1d1e3c3, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.070076 1264 consensus_queue.cc:260] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.070200 1265 tablet_bootstrap.cc:654] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.070382 1264 ts_tablet_manager.cc:1398] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:43.071000 1264 tablet_bootstrap.cc:492] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3: Bootstrap starting.
I20240502 07:59:43.071935 1264 tablet_bootstrap.cc:654] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.072388 1266 tablet_bootstrap.cc:492] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f: No bootstrap required, opened a new log
I20240502 07:59:43.072492 1266 ts_tablet_manager.cc:1367] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.001s
I20240502 07:59:43.072935 1266 raft_consensus.cc:348] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.072979 1265 tablet_bootstrap.cc:492] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70: No bootstrap required, opened a new log
I20240502 07:59:43.073025 1266 raft_consensus.cc:374] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.073064 1266 raft_consensus.cc:729] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27ec1b26c2f64cc682ed7ae12e133a9f, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.073103 1265 ts_tablet_manager.cc:1367] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:59:43.073191 1266 consensus_queue.cc:260] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.073565 1266 ts_tablet_manager.cc:1398] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:43.073673 1265 raft_consensus.cc:348] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.073735 1266 tablet_bootstrap.cc:492] T fda98cfe14f947969fb41e26c75bc384 P 27ec1b26c2f64cc682ed7ae12e133a9f: Bootstrap starting.
I20240502 07:59:43.073787 1265 raft_consensus.cc:374] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.073850 1265 raft_consensus.cc:729] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e9945160714b16b2db91f90c5ccc70, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.073987 1265 consensus_queue.cc:260] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.074455 1265 ts_tablet_manager.cc:1398] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:59:43.074923 1265 tablet_bootstrap.cc:492] T fda98cfe14f947969fb41e26c75bc384 P b1e9945160714b16b2db91f90c5ccc70: Bootstrap starting.
I20240502 07:59:43.074973 1266 tablet_bootstrap.cc:654] T fda98cfe14f947969fb41e26c75bc384 P 27ec1b26c2f64cc682ed7ae12e133a9f: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.075994 1265 tablet_bootstrap.cc:654] T fda98cfe14f947969fb41e26c75bc384 P b1e9945160714b16b2db91f90c5ccc70: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.076510 1264 tablet_bootstrap.cc:492] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3: No bootstrap required, opened a new log
I20240502 07:59:43.076627 1264 ts_tablet_manager.cc:1367] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240502 07:59:43.077116 1264 raft_consensus.cc:348] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.077195 1266 tablet_bootstrap.cc:492] T fda98cfe14f947969fb41e26c75bc384 P 27ec1b26c2f64cc682ed7ae12e133a9f: No bootstrap required, opened a new log
I20240502 07:59:43.077251 1264 raft_consensus.cc:374] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.077292 1264 raft_consensus.cc:729] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e648af9dbb46fc881340efe1d1e3c3, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.077318 1266 ts_tablet_manager.cc:1367] T fda98cfe14f947969fb41e26c75bc384 P 27ec1b26c2f64cc682ed7ae12e133a9f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240502 07:59:43.077469 1264 consensus_queue.cc:260] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.077749 1266 raft_consensus.cc:348] T fda98cfe14f947969fb41e26c75bc384 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.077867 1266 raft_consensus.cc:374] T fda98cfe14f947969fb41e26c75bc384 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.077801 1264 ts_tablet_manager.cc:1398] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:43.077962 1265 tablet_bootstrap.cc:492] T fda98cfe14f947969fb41e26c75bc384 P b1e9945160714b16b2db91f90c5ccc70: No bootstrap required, opened a new log
I20240502 07:59:43.078056 1265 ts_tablet_manager.cc:1367] T fda98cfe14f947969fb41e26c75bc384 P b1e9945160714b16b2db91f90c5ccc70: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:43.078109 1264 tablet_bootstrap.cc:492] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3: Bootstrap starting.
I20240502 07:59:43.077912 1266 raft_consensus.cc:729] T fda98cfe14f947969fb41e26c75bc384 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27ec1b26c2f64cc682ed7ae12e133a9f, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.078532 1266 consensus_queue.cc:260] T fda98cfe14f947969fb41e26c75bc384 P 27ec1b26c2f64cc682ed7ae12e133a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.078575 1265 raft_consensus.cc:348] T fda98cfe14f947969fb41e26c75bc384 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.078680 1265 raft_consensus.cc:374] T fda98cfe14f947969fb41e26c75bc384 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.078764 1265 raft_consensus.cc:729] T fda98cfe14f947969fb41e26c75bc384 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e9945160714b16b2db91f90c5ccc70, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.078866 1266 ts_tablet_manager.cc:1398] T fda98cfe14f947969fb41e26c75bc384 P 27ec1b26c2f64cc682ed7ae12e133a9f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:43.079034 1266 tablet_bootstrap.cc:492] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f: Bootstrap starting.
I20240502 07:59:43.079280 1264 tablet_bootstrap.cc:654] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.080121 1266 tablet_bootstrap.cc:654] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.078917 1265 consensus_queue.cc:260] T fda98cfe14f947969fb41e26c75bc384 P b1e9945160714b16b2db91f90c5ccc70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.080690 1265 ts_tablet_manager.cc:1398] T fda98cfe14f947969fb41e26c75bc384 P b1e9945160714b16b2db91f90c5ccc70: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:43.080897 1265 tablet_bootstrap.cc:492] T 58c5f52a420c467684b2cb50972e7c99 P b1e9945160714b16b2db91f90c5ccc70: Bootstrap starting.
I20240502 07:59:43.081323 1264 tablet_bootstrap.cc:492] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3: No bootstrap required, opened a new log
I20240502 07:59:43.081419 1264 ts_tablet_manager.cc:1367] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 07:59:43.082015 1265 tablet_bootstrap.cc:654] T 58c5f52a420c467684b2cb50972e7c99 P b1e9945160714b16b2db91f90c5ccc70: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.081955 1264 raft_consensus.cc:348] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.082077 1264 raft_consensus.cc:374] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.082139 1264 raft_consensus.cc:729] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e648af9dbb46fc881340efe1d1e3c3, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.082257 1264 consensus_queue.cc:260] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.083168 1264 ts_tablet_manager.cc:1398] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:43.083173 1266 tablet_bootstrap.cc:492] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f: No bootstrap required, opened a new log
I20240502 07:59:43.083407 1264 tablet_bootstrap.cc:492] T 6edf17550e3f4a0b945477edc6de6af1 P 77e648af9dbb46fc881340efe1d1e3c3: Bootstrap starting.
I20240502 07:59:43.083413 1266 ts_tablet_manager.cc:1367] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 07:59:43.083972 1265 tablet_bootstrap.cc:492] T 58c5f52a420c467684b2cb50972e7c99 P b1e9945160714b16b2db91f90c5ccc70: No bootstrap required, opened a new log
I20240502 07:59:43.084064 1265 ts_tablet_manager.cc:1367] T 58c5f52a420c467684b2cb50972e7c99 P b1e9945160714b16b2db91f90c5ccc70: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:43.083999 1266 raft_consensus.cc:348] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.084110 1266 raft_consensus.cc:374] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.084146 1266 raft_consensus.cc:729] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27ec1b26c2f64cc682ed7ae12e133a9f, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.084347 1266 consensus_queue.cc:260] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.084544 1265 raft_consensus.cc:348] T 58c5f52a420c467684b2cb50972e7c99 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.084695 1265 raft_consensus.cc:374] T 58c5f52a420c467684b2cb50972e7c99 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.084751 1265 raft_consensus.cc:729] T 58c5f52a420c467684b2cb50972e7c99 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e9945160714b16b2db91f90c5ccc70, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.084887 1265 consensus_queue.cc:260] T 58c5f52a420c467684b2cb50972e7c99 P b1e9945160714b16b2db91f90c5ccc70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.085186 1265 ts_tablet_manager.cc:1398] T 58c5f52a420c467684b2cb50972e7c99 P b1e9945160714b16b2db91f90c5ccc70: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:43.084528 1264 tablet_bootstrap.cc:654] T 6edf17550e3f4a0b945477edc6de6af1 P 77e648af9dbb46fc881340efe1d1e3c3: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.085395 1265 tablet_bootstrap.cc:492] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70: Bootstrap starting.
I20240502 07:59:43.086421 1265 tablet_bootstrap.cc:654] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.086668 1264 tablet_bootstrap.cc:492] T 6edf17550e3f4a0b945477edc6de6af1 P 77e648af9dbb46fc881340efe1d1e3c3: No bootstrap required, opened a new log
I20240502 07:59:43.086833 1264 ts_tablet_manager.cc:1367] T 6edf17550e3f4a0b945477edc6de6af1 P 77e648af9dbb46fc881340efe1d1e3c3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:43.085172 1266 ts_tablet_manager.cc:1398] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 07:59:43.087193 1266 tablet_bootstrap.cc:492] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f: Bootstrap starting.
I20240502 07:59:43.087325 1264 raft_consensus.cc:348] T 6edf17550e3f4a0b945477edc6de6af1 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.087435 1264 raft_consensus.cc:374] T 6edf17550e3f4a0b945477edc6de6af1 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.087481 1264 raft_consensus.cc:729] T 6edf17550e3f4a0b945477edc6de6af1 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e648af9dbb46fc881340efe1d1e3c3, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.087576 1265 tablet_bootstrap.cc:492] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70: No bootstrap required, opened a new log
I20240502 07:59:43.087661 1265 ts_tablet_manager.cc:1367] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:43.087615 1264 consensus_queue.cc:260] T 6edf17550e3f4a0b945477edc6de6af1 P 77e648af9dbb46fc881340efe1d1e3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.088061 1265 raft_consensus.cc:348] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.088181 1265 raft_consensus.cc:374] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.088220 1265 raft_consensus.cc:729] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e9945160714b16b2db91f90c5ccc70, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.088230 1266 tablet_bootstrap.cc:654] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.088493 1265 consensus_queue.cc:260] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.088619 1277 raft_consensus.cc:482] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:43.088707 1264 ts_tablet_manager.cc:1398] T 6edf17550e3f4a0b945477edc6de6af1 P 77e648af9dbb46fc881340efe1d1e3c3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:43.088681 1277 raft_consensus.cc:504] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.088877 1264 tablet_bootstrap.cc:492] T 5437f94679a841e3980053e69d624d76 P 77e648af9dbb46fc881340efe1d1e3c3: Bootstrap starting.
I20240502 07:59:43.089175 1277 leader_election.cc:288] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761), 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:43.090099 1264 tablet_bootstrap.cc:654] T 5437f94679a841e3980053e69d624d76 P 77e648af9dbb46fc881340efe1d1e3c3: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.090200 1265 ts_tablet_manager.cc:1398] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:43.090400 1265 tablet_bootstrap.cc:492] T c7c1536618ae4e74aa79349d04201fb9 P b1e9945160714b16b2db91f90c5ccc70: Bootstrap starting.
I20240502 07:59:43.091194 1264 tablet_bootstrap.cc:492] T 5437f94679a841e3980053e69d624d76 P 77e648af9dbb46fc881340efe1d1e3c3: No bootstrap required, opened a new log
I20240502 07:59:43.091297 1264 ts_tablet_manager.cc:1367] T 5437f94679a841e3980053e69d624d76 P 77e648af9dbb46fc881340efe1d1e3c3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:43.091470 1265 tablet_bootstrap.cc:654] T c7c1536618ae4e74aa79349d04201fb9 P b1e9945160714b16b2db91f90c5ccc70: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.091738 1264 raft_consensus.cc:348] T 5437f94679a841e3980053e69d624d76 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.091856 1264 raft_consensus.cc:374] T 5437f94679a841e3980053e69d624d76 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.091892 1264 raft_consensus.cc:729] T 5437f94679a841e3980053e69d624d76 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e648af9dbb46fc881340efe1d1e3c3, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.092049 1264 consensus_queue.cc:260] T 5437f94679a841e3980053e69d624d76 P 77e648af9dbb46fc881340efe1d1e3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.092371 1264 ts_tablet_manager.cc:1398] T 5437f94679a841e3980053e69d624d76 P 77e648af9dbb46fc881340efe1d1e3c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:43.092638 1264 tablet_bootstrap.cc:492] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3: Bootstrap starting.
I20240502 07:59:43.093631 1264 tablet_bootstrap.cc:654] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.094763 1266 tablet_bootstrap.cc:492] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f: No bootstrap required, opened a new log
I20240502 07:59:43.094949 1266 ts_tablet_manager.cc:1367] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240502 07:59:43.095186 1264 tablet_bootstrap.cc:492] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3: No bootstrap required, opened a new log
I20240502 07:59:43.095270 1264 ts_tablet_manager.cc:1367] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:43.095556 1266 raft_consensus.cc:348] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.095626 1264 raft_consensus.cc:348] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.095757 1264 raft_consensus.cc:374] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.095989 1264 raft_consensus.cc:729] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e648af9dbb46fc881340efe1d1e3c3, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.096109 1264 consensus_queue.cc:260] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.095743 1266 raft_consensus.cc:374] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.096472 1266 raft_consensus.cc:729] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27ec1b26c2f64cc682ed7ae12e133a9f, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.096597 1067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5437f94679a841e3980053e69d624d76" candidate_uuid: "b1e9945160714b16b2db91f90c5ccc70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3" is_pre_election: true
I20240502 07:59:43.096717 1067 raft_consensus.cc:2454] T 5437f94679a841e3980053e69d624d76 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1e9945160714b16b2db91f90c5ccc70 in term 0.
I20240502 07:59:43.096642 1266 consensus_queue.cc:260] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.097126 1265 tablet_bootstrap.cc:492] T c7c1536618ae4e74aa79349d04201fb9 P b1e9945160714b16b2db91f90c5ccc70: No bootstrap required, opened a new log
I20240502 07:59:43.097213 1107 leader_election.cc:302] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e648af9dbb46fc881340efe1d1e3c3, b1e9945160714b16b2db91f90c5ccc70; no voters:
I20240502 07:59:43.096428 1264 ts_tablet_manager.cc:1398] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:43.097388 1277 raft_consensus.cc:2781] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:43.097525 1264 tablet_bootstrap.cc:492] T 2c1b6523d6724a02acc9ae19f3f0255e P 77e648af9dbb46fc881340efe1d1e3c3: Bootstrap starting.
I20240502 07:59:43.097532 1277 raft_consensus.cc:482] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:43.097661 1277 raft_consensus.cc:3037] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.098498 1264 tablet_bootstrap.cc:654] T 2c1b6523d6724a02acc9ae19f3f0255e P 77e648af9dbb46fc881340efe1d1e3c3: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.098747 1277 raft_consensus.cc:504] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.097259 1265 ts_tablet_manager.cc:1367] T c7c1536618ae4e74aa79349d04201fb9 P b1e9945160714b16b2db91f90c5ccc70: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 07:59:43.099125 1277 leader_election.cc:288] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [CANDIDATE]: Term 1 election: Requested vote from peers 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761), 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:43.099484 1265 raft_consensus.cc:348] T c7c1536618ae4e74aa79349d04201fb9 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.099586 1265 raft_consensus.cc:374] T c7c1536618ae4e74aa79349d04201fb9 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.099622 1265 raft_consensus.cc:729] T c7c1536618ae4e74aa79349d04201fb9 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e9945160714b16b2db91f90c5ccc70, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.099793 1067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5437f94679a841e3980053e69d624d76" candidate_uuid: "b1e9945160714b16b2db91f90c5ccc70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3"
I20240502 07:59:43.099777 1265 consensus_queue.cc:260] T c7c1536618ae4e74aa79349d04201fb9 P b1e9945160714b16b2db91f90c5ccc70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.099905 1067 raft_consensus.cc:3037] T 5437f94679a841e3980053e69d624d76 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.099931 1264 tablet_bootstrap.cc:492] T 2c1b6523d6724a02acc9ae19f3f0255e P 77e648af9dbb46fc881340efe1d1e3c3: No bootstrap required, opened a new log
I20240502 07:59:43.100031 1264 ts_tablet_manager.cc:1367] T 2c1b6523d6724a02acc9ae19f3f0255e P 77e648af9dbb46fc881340efe1d1e3c3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:43.100596 1264 raft_consensus.cc:348] T 2c1b6523d6724a02acc9ae19f3f0255e P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.100733 1264 raft_consensus.cc:374] T 2c1b6523d6724a02acc9ae19f3f0255e P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.100768 1264 raft_consensus.cc:729] T 2c1b6523d6724a02acc9ae19f3f0255e P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e648af9dbb46fc881340efe1d1e3c3, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.100914 1264 consensus_queue.cc:260] T 2c1b6523d6724a02acc9ae19f3f0255e P 77e648af9dbb46fc881340efe1d1e3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.100924 1067 raft_consensus.cc:2454] T 5437f94679a841e3980053e69d624d76 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1e9945160714b16b2db91f90c5ccc70 in term 1.
I20240502 07:59:43.101516 1107 leader_election.cc:302] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e648af9dbb46fc881340efe1d1e3c3, b1e9945160714b16b2db91f90c5ccc70; no voters:
I20240502 07:59:43.097345 1266 ts_tablet_manager.cc:1398] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 07:59:43.101696 1277 raft_consensus.cc:2781] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:43.101796 1218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5437f94679a841e3980053e69d624d76" candidate_uuid: "b1e9945160714b16b2db91f90c5ccc70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" is_pre_election: true
I20240502 07:59:43.101891 1266 tablet_bootstrap.cc:492] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f: Bootstrap starting.
I20240502 07:59:43.102003 1217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5437f94679a841e3980053e69d624d76" candidate_uuid: "b1e9945160714b16b2db91f90c5ccc70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
I20240502 07:59:43.101711 1264 ts_tablet_manager.cc:1398] T 2c1b6523d6724a02acc9ae19f3f0255e P 77e648af9dbb46fc881340efe1d1e3c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:43.102103 1265 ts_tablet_manager.cc:1398] T c7c1536618ae4e74aa79349d04201fb9 P b1e9945160714b16b2db91f90c5ccc70: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 07:59:43.102244 1264 tablet_bootstrap.cc:492] T cdf0ff2967c040a589f915172d3f446b P 77e648af9dbb46fc881340efe1d1e3c3: Bootstrap starting.
I20240502 07:59:43.102280 1265 tablet_bootstrap.cc:492] T 69a9ebd0c390436b8ace08089f719358 P b1e9945160714b16b2db91f90c5ccc70: Bootstrap starting.
I20240502 07:59:43.101902 1218 raft_consensus.cc:2454] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1e9945160714b16b2db91f90c5ccc70 in term 0.
I20240502 07:59:43.103160 1266 tablet_bootstrap.cc:654] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.103322 1264 tablet_bootstrap.cc:654] T cdf0ff2967c040a589f915172d3f446b P 77e648af9dbb46fc881340efe1d1e3c3: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.103443 1265 tablet_bootstrap.cc:654] T 69a9ebd0c390436b8ace08089f719358 P b1e9945160714b16b2db91f90c5ccc70: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.101809 1277 raft_consensus.cc:686] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Becoming Leader. State: Replica: b1e9945160714b16b2db91f90c5ccc70, State: Running, Role: LEADER
I20240502 07:59:43.104302 1277 consensus_queue.cc:237] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.105036 1264 tablet_bootstrap.cc:492] T cdf0ff2967c040a589f915172d3f446b P 77e648af9dbb46fc881340efe1d1e3c3: No bootstrap required, opened a new log
I20240502 07:59:43.105123 1264 ts_tablet_manager.cc:1367] T cdf0ff2967c040a589f915172d3f446b P 77e648af9dbb46fc881340efe1d1e3c3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:43.105789 1264 raft_consensus.cc:348] T cdf0ff2967c040a589f915172d3f446b P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.105911 1264 raft_consensus.cc:374] T cdf0ff2967c040a589f915172d3f446b P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.105957 1264 raft_consensus.cc:729] T cdf0ff2967c040a589f915172d3f446b P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e648af9dbb46fc881340efe1d1e3c3, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.106063 1264 consensus_queue.cc:260] T cdf0ff2967c040a589f915172d3f446b P 77e648af9dbb46fc881340efe1d1e3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.106230 963 catalog_manager.cc:5484] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130). New cstate: current_term: 1 leader_uuid: "b1e9945160714b16b2db91f90c5ccc70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:43.106369 1264 ts_tablet_manager.cc:1398] T cdf0ff2967c040a589f915172d3f446b P 77e648af9dbb46fc881340efe1d1e3c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:43.106518 1265 tablet_bootstrap.cc:492] T 69a9ebd0c390436b8ace08089f719358 P b1e9945160714b16b2db91f90c5ccc70: No bootstrap required, opened a new log
I20240502 07:59:43.106534 1264 tablet_bootstrap.cc:492] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3: Bootstrap starting.
I20240502 07:59:43.106626 1265 ts_tablet_manager.cc:1367] T 69a9ebd0c390436b8ace08089f719358 P b1e9945160714b16b2db91f90c5ccc70: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:59:43.107151 1265 raft_consensus.cc:348] T 69a9ebd0c390436b8ace08089f719358 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.107263 1265 raft_consensus.cc:374] T 69a9ebd0c390436b8ace08089f719358 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.107298 1265 raft_consensus.cc:729] T 69a9ebd0c390436b8ace08089f719358 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e9945160714b16b2db91f90c5ccc70, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.107445 1265 consensus_queue.cc:260] T 69a9ebd0c390436b8ace08089f719358 P b1e9945160714b16b2db91f90c5ccc70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.107523 1264 tablet_bootstrap.cc:654] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.107828 1265 ts_tablet_manager.cc:1398] T 69a9ebd0c390436b8ace08089f719358 P b1e9945160714b16b2db91f90c5ccc70: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:43.108024 1265 tablet_bootstrap.cc:492] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70: Bootstrap starting.
I20240502 07:59:43.108865 1264 tablet_bootstrap.cc:492] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3: No bootstrap required, opened a new log
I20240502 07:59:43.108954 1264 ts_tablet_manager.cc:1367] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:43.109151 1265 tablet_bootstrap.cc:654] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.109411 1264 raft_consensus.cc:348] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.109541 1264 raft_consensus.cc:374] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.109589 1264 raft_consensus.cc:729] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e648af9dbb46fc881340efe1d1e3c3, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.109700 1264 consensus_queue.cc:260] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.110029 1264 ts_tablet_manager.cc:1398] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:43.110196 1266 tablet_bootstrap.cc:492] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f: No bootstrap required, opened a new log
I20240502 07:59:43.110239 1264 tablet_bootstrap.cc:492] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3: Bootstrap starting.
I20240502 07:59:43.110291 1266 ts_tablet_manager.cc:1367] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240502 07:59:43.110791 1266 raft_consensus.cc:348] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.110901 1266 raft_consensus.cc:374] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.110939 1266 raft_consensus.cc:729] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27ec1b26c2f64cc682ed7ae12e133a9f, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.111088 1266 consensus_queue.cc:260] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.111307 1264 tablet_bootstrap.cc:654] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.111392 1266 ts_tablet_manager.cc:1398] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:43.111577 1265 tablet_bootstrap.cc:492] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70: No bootstrap required, opened a new log
I20240502 07:59:43.111711 1265 ts_tablet_manager.cc:1367] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:59:43.112143 1265 raft_consensus.cc:348] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.112259 1265 raft_consensus.cc:374] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.112319 1265 raft_consensus.cc:729] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e9945160714b16b2db91f90c5ccc70, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.112433 1265 consensus_queue.cc:260] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.112720 1265 ts_tablet_manager.cc:1398] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:43.112911 1265 tablet_bootstrap.cc:492] T df4233b549b34b01b6f9662c7e7b3515 P b1e9945160714b16b2db91f90c5ccc70: Bootstrap starting.
I20240502 07:59:43.113577 1264 tablet_bootstrap.cc:492] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3: No bootstrap required, opened a new log
I20240502 07:59:43.113670 1264 ts_tablet_manager.cc:1367] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:43.113898 1265 tablet_bootstrap.cc:654] T df4233b549b34b01b6f9662c7e7b3515 P b1e9945160714b16b2db91f90c5ccc70: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.114106 1264 raft_consensus.cc:348] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.114214 1264 raft_consensus.cc:374] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.111560 1266 tablet_bootstrap.cc:492] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f: Bootstrap starting.
I20240502 07:59:43.114257 1264 raft_consensus.cc:729] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e648af9dbb46fc881340efe1d1e3c3, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.114413 1264 consensus_queue.cc:260] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.114738 1264 ts_tablet_manager.cc:1398] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:43.114919 1264 tablet_bootstrap.cc:492] T 682e26b333104f82863b92c545cfdafe P 77e648af9dbb46fc881340efe1d1e3c3: Bootstrap starting.
I20240502 07:59:43.115224 1266 tablet_bootstrap.cc:654] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.115830 1264 tablet_bootstrap.cc:654] T 682e26b333104f82863b92c545cfdafe P 77e648af9dbb46fc881340efe1d1e3c3: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.116650 1266 tablet_bootstrap.cc:492] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f: No bootstrap required, opened a new log
I20240502 07:59:43.116762 1266 ts_tablet_manager.cc:1367] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240502 07:59:43.117209 1264 tablet_bootstrap.cc:492] T 682e26b333104f82863b92c545cfdafe P 77e648af9dbb46fc881340efe1d1e3c3: No bootstrap required, opened a new log
I20240502 07:59:43.117247 1265 tablet_bootstrap.cc:492] T df4233b549b34b01b6f9662c7e7b3515 P b1e9945160714b16b2db91f90c5ccc70: No bootstrap required, opened a new log
I20240502 07:59:43.117286 1266 raft_consensus.cc:348] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.117539 1265 ts_tablet_manager.cc:1367] T df4233b549b34b01b6f9662c7e7b3515 P b1e9945160714b16b2db91f90c5ccc70: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 07:59:43.117493 1264 ts_tablet_manager.cc:1367] T 682e26b333104f82863b92c545cfdafe P 77e648af9dbb46fc881340efe1d1e3c3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 07:59:43.117870 1266 raft_consensus.cc:374] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.117931 1266 raft_consensus.cc:729] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27ec1b26c2f64cc682ed7ae12e133a9f, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.117980 1265 raft_consensus.cc:348] T df4233b549b34b01b6f9662c7e7b3515 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.118081 1265 raft_consensus.cc:374] T df4233b549b34b01b6f9662c7e7b3515 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.118196 1265 raft_consensus.cc:729] T df4233b549b34b01b6f9662c7e7b3515 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e9945160714b16b2db91f90c5ccc70, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.118088 1266 consensus_queue.cc:260] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.118340 1265 consensus_queue.cc:260] T df4233b549b34b01b6f9662c7e7b3515 P b1e9945160714b16b2db91f90c5ccc70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.118098 1264 raft_consensus.cc:348] T 682e26b333104f82863b92c545cfdafe P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.118465 1264 raft_consensus.cc:374] T 682e26b333104f82863b92c545cfdafe P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.118525 1264 raft_consensus.cc:729] T 682e26b333104f82863b92c545cfdafe P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e648af9dbb46fc881340efe1d1e3c3, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.118662 1264 consensus_queue.cc:260] T 682e26b333104f82863b92c545cfdafe P 77e648af9dbb46fc881340efe1d1e3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.118965 1265 ts_tablet_manager.cc:1398] T df4233b549b34b01b6f9662c7e7b3515 P b1e9945160714b16b2db91f90c5ccc70: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:43.118999 1264 ts_tablet_manager.cc:1398] T 682e26b333104f82863b92c545cfdafe P 77e648af9dbb46fc881340efe1d1e3c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:43.119134 1265 tablet_bootstrap.cc:492] T d0d982dd52d246348fb3984fd1c3a40f P b1e9945160714b16b2db91f90c5ccc70: Bootstrap starting.
I20240502 07:59:43.120172 1265 tablet_bootstrap.cc:654] T d0d982dd52d246348fb3984fd1c3a40f P b1e9945160714b16b2db91f90c5ccc70: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.120236 1266 ts_tablet_manager.cc:1398] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 07:59:43.121166 1266 tablet_bootstrap.cc:492] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f: Bootstrap starting.
I20240502 07:59:43.122068 1265 tablet_bootstrap.cc:492] T d0d982dd52d246348fb3984fd1c3a40f P b1e9945160714b16b2db91f90c5ccc70: No bootstrap required, opened a new log
I20240502 07:59:43.122174 1265 ts_tablet_manager.cc:1367] T d0d982dd52d246348fb3984fd1c3a40f P b1e9945160714b16b2db91f90c5ccc70: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:43.122431 1266 tablet_bootstrap.cc:654] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.122674 1265 raft_consensus.cc:348] T d0d982dd52d246348fb3984fd1c3a40f P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.122823 1265 raft_consensus.cc:374] T d0d982dd52d246348fb3984fd1c3a40f P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.122859 1265 raft_consensus.cc:729] T d0d982dd52d246348fb3984fd1c3a40f P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e9945160714b16b2db91f90c5ccc70, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.123029 1265 consensus_queue.cc:260] T d0d982dd52d246348fb3984fd1c3a40f P b1e9945160714b16b2db91f90c5ccc70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.123313 1265 ts_tablet_manager.cc:1398] T d0d982dd52d246348fb3984fd1c3a40f P b1e9945160714b16b2db91f90c5ccc70: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:43.123494 1265 tablet_bootstrap.cc:492] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70: Bootstrap starting.
I20240502 07:59:43.123828 1266 tablet_bootstrap.cc:492] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f: No bootstrap required, opened a new log
I20240502 07:59:43.123932 1266 ts_tablet_manager.cc:1367] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:43.124444 1265 tablet_bootstrap.cc:654] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.124397 1266 raft_consensus.cc:348] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.124503 1266 raft_consensus.cc:374] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.124547 1266 raft_consensus.cc:729] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27ec1b26c2f64cc682ed7ae12e133a9f, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.124717 1266 consensus_queue.cc:260] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.125514 1266 ts_tablet_manager.cc:1398] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:43.125514 1265 tablet_bootstrap.cc:492] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70: No bootstrap required, opened a new log
I20240502 07:59:43.125651 1265 ts_tablet_manager.cc:1367] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:43.125725 1266 tablet_bootstrap.cc:492] T cdf0ff2967c040a589f915172d3f446b P 27ec1b26c2f64cc682ed7ae12e133a9f: Bootstrap starting.
I20240502 07:59:43.126042 1265 raft_consensus.cc:348] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.126157 1265 raft_consensus.cc:374] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.126189 1265 raft_consensus.cc:729] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e9945160714b16b2db91f90c5ccc70, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.126315 1265 consensus_queue.cc:260] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.126605 1265 ts_tablet_manager.cc:1398] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:43.126646 1266 tablet_bootstrap.cc:654] T cdf0ff2967c040a589f915172d3f446b P 27ec1b26c2f64cc682ed7ae12e133a9f: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.127676 1266 tablet_bootstrap.cc:492] T cdf0ff2967c040a589f915172d3f446b P 27ec1b26c2f64cc682ed7ae12e133a9f: No bootstrap required, opened a new log
I20240502 07:59:43.127769 1266 ts_tablet_manager.cc:1367] T cdf0ff2967c040a589f915172d3f446b P 27ec1b26c2f64cc682ed7ae12e133a9f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:43.128227 1266 raft_consensus.cc:348] T cdf0ff2967c040a589f915172d3f446b P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.128365 1266 raft_consensus.cc:374] T cdf0ff2967c040a589f915172d3f446b P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.128402 1266 raft_consensus.cc:729] T cdf0ff2967c040a589f915172d3f446b P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27ec1b26c2f64cc682ed7ae12e133a9f, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.128531 1266 consensus_queue.cc:260] T cdf0ff2967c040a589f915172d3f446b P 27ec1b26c2f64cc682ed7ae12e133a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.128803 1266 ts_tablet_manager.cc:1398] T cdf0ff2967c040a589f915172d3f446b P 27ec1b26c2f64cc682ed7ae12e133a9f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:43.129017 1266 tablet_bootstrap.cc:492] T c7c1536618ae4e74aa79349d04201fb9 P 27ec1b26c2f64cc682ed7ae12e133a9f: Bootstrap starting.
I20240502 07:59:43.129881 1266 tablet_bootstrap.cc:654] T c7c1536618ae4e74aa79349d04201fb9 P 27ec1b26c2f64cc682ed7ae12e133a9f: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.131101 1266 tablet_bootstrap.cc:492] T c7c1536618ae4e74aa79349d04201fb9 P 27ec1b26c2f64cc682ed7ae12e133a9f: No bootstrap required, opened a new log
I20240502 07:59:43.131178 1266 ts_tablet_manager.cc:1367] T c7c1536618ae4e74aa79349d04201fb9 P 27ec1b26c2f64cc682ed7ae12e133a9f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:43.131498 1269 raft_consensus.cc:482] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:43.131616 1266 raft_consensus.cc:348] T c7c1536618ae4e74aa79349d04201fb9 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.131644 1269 raft_consensus.cc:504] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.131739 1266 raft_consensus.cc:374] T c7c1536618ae4e74aa79349d04201fb9 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.131840 1266 raft_consensus.cc:729] T c7c1536618ae4e74aa79349d04201fb9 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27ec1b26c2f64cc682ed7ae12e133a9f, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.131978 1266 consensus_queue.cc:260] T c7c1536618ae4e74aa79349d04201fb9 P 27ec1b26c2f64cc682ed7ae12e133a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.132304 1269 leader_election.cc:288] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203), 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:43.132367 1266 ts_tablet_manager.cc:1398] T c7c1536618ae4e74aa79349d04201fb9 P 27ec1b26c2f64cc682ed7ae12e133a9f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:43.132547 1266 tablet_bootstrap.cc:492] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f: Bootstrap starting.
I20240502 07:59:43.133551 1266 tablet_bootstrap.cc:654] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.135589 1266 tablet_bootstrap.cc:492] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f: No bootstrap required, opened a new log
I20240502 07:59:43.135681 1266 ts_tablet_manager.cc:1367] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:43.136054 1266 raft_consensus.cc:348] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.136152 1266 raft_consensus.cc:374] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.136190 1266 raft_consensus.cc:729] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27ec1b26c2f64cc682ed7ae12e133a9f, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.136351 1266 consensus_queue.cc:260] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.136627 1266 ts_tablet_manager.cc:1398] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:43.136818 1266 tablet_bootstrap.cc:492] T df4233b549b34b01b6f9662c7e7b3515 P 27ec1b26c2f64cc682ed7ae12e133a9f: Bootstrap starting.
I20240502 07:59:43.137754 1266 tablet_bootstrap.cc:654] T df4233b549b34b01b6f9662c7e7b3515 P 27ec1b26c2f64cc682ed7ae12e133a9f: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.137769 1143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fda98cfe14f947969fb41e26c75bc384" candidate_uuid: "77e648af9dbb46fc881340efe1d1e3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e9945160714b16b2db91f90c5ccc70" is_pre_election: true
I20240502 07:59:43.137876 1143 raft_consensus.cc:2454] T fda98cfe14f947969fb41e26c75bc384 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e648af9dbb46fc881340efe1d1e3c3 in term 0.
I20240502 07:59:43.138235 1030 leader_election.cc:302] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e648af9dbb46fc881340efe1d1e3c3, b1e9945160714b16b2db91f90c5ccc70; no voters:
I20240502 07:59:43.138480 1269 raft_consensus.cc:2781] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:43.138550 1269 raft_consensus.cc:482] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:43.138579 1269 raft_consensus.cc:3037] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.139189 1266 tablet_bootstrap.cc:492] T df4233b549b34b01b6f9662c7e7b3515 P 27ec1b26c2f64cc682ed7ae12e133a9f: No bootstrap required, opened a new log
I20240502 07:59:43.139278 1266 ts_tablet_manager.cc:1367] T df4233b549b34b01b6f9662c7e7b3515 P 27ec1b26c2f64cc682ed7ae12e133a9f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:43.139647 1266 raft_consensus.cc:348] T df4233b549b34b01b6f9662c7e7b3515 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.139734 1266 raft_consensus.cc:374] T df4233b549b34b01b6f9662c7e7b3515 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.139768 1266 raft_consensus.cc:729] T df4233b549b34b01b6f9662c7e7b3515 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27ec1b26c2f64cc682ed7ae12e133a9f, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.139866 1269 raft_consensus.cc:504] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.139909 1266 consensus_queue.cc:260] T df4233b549b34b01b6f9662c7e7b3515 P 27ec1b26c2f64cc682ed7ae12e133a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.140321 1269 leader_election.cc:288] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [CANDIDATE]: Term 1 election: Requested vote from peers b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203), 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:43.140650 1143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fda98cfe14f947969fb41e26c75bc384" candidate_uuid: "77e648af9dbb46fc881340efe1d1e3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
I20240502 07:59:43.140686 1266 ts_tablet_manager.cc:1398] T df4233b549b34b01b6f9662c7e7b3515 P 27ec1b26c2f64cc682ed7ae12e133a9f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:43.140781 1143 raft_consensus.cc:3037] T fda98cfe14f947969fb41e26c75bc384 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.140878 1266 tablet_bootstrap.cc:492] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f: Bootstrap starting.
I20240502 07:59:43.141609 1143 raft_consensus.cc:2454] T fda98cfe14f947969fb41e26c75bc384 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e648af9dbb46fc881340efe1d1e3c3 in term 1.
I20240502 07:59:43.141899 1266 tablet_bootstrap.cc:654] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:43.141912 1030 leader_election.cc:302] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e648af9dbb46fc881340efe1d1e3c3, b1e9945160714b16b2db91f90c5ccc70; no voters:
I20240502 07:59:43.142114 1269 raft_consensus.cc:2781] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:43.142383 1269 raft_consensus.cc:686] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Becoming Leader. State: Replica: 77e648af9dbb46fc881340efe1d1e3c3, State: Running, Role: LEADER
I20240502 07:59:43.142547 1269 consensus_queue.cc:237] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.144163 1218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fda98cfe14f947969fb41e26c75bc384" candidate_uuid: "77e648af9dbb46fc881340efe1d1e3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" is_pre_election: true
I20240502 07:59:43.144177 1217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fda98cfe14f947969fb41e26c75bc384" candidate_uuid: "77e648af9dbb46fc881340efe1d1e3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
I20240502 07:59:43.144364 1218 raft_consensus.cc:2454] T fda98cfe14f947969fb41e26c75bc384 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e648af9dbb46fc881340efe1d1e3c3 in term 0.
I20240502 07:59:43.144573 963 catalog_manager.cc:5484] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129). New cstate: current_term: 1 leader_uuid: "77e648af9dbb46fc881340efe1d1e3c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:43.145675 1266 tablet_bootstrap.cc:492] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f: No bootstrap required, opened a new log
I20240502 07:59:43.145835 1266 ts_tablet_manager.cc:1367] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 07:59:43.146317 1266 raft_consensus.cc:348] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.146420 1266 raft_consensus.cc:374] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:43.146453 1266 raft_consensus.cc:729] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27ec1b26c2f64cc682ed7ae12e133a9f, State: Initialized, Role: FOLLOWER
I20240502 07:59:43.146596 1266 consensus_queue.cc:260] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.146905 1266 ts_tablet_manager.cc:1398] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:43.170256 1277 raft_consensus.cc:482] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:43.170404 1277 raft_consensus.cc:504] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.170847 1277 leader_election.cc:288] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761), 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:43.171203 1067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6edf17550e3f4a0b945477edc6de6af1" candidate_uuid: "b1e9945160714b16b2db91f90c5ccc70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3" is_pre_election: true
I20240502 07:59:43.171252 1218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6edf17550e3f4a0b945477edc6de6af1" candidate_uuid: "b1e9945160714b16b2db91f90c5ccc70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" is_pre_election: true
I20240502 07:59:43.171355 1218 raft_consensus.cc:2454] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1e9945160714b16b2db91f90c5ccc70 in term 0.
I20240502 07:59:43.171355 1067 raft_consensus.cc:2454] T 6edf17550e3f4a0b945477edc6de6af1 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1e9945160714b16b2db91f90c5ccc70 in term 0.
I20240502 07:59:43.171659 1107 leader_election.cc:302] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e648af9dbb46fc881340efe1d1e3c3, b1e9945160714b16b2db91f90c5ccc70; no voters:
I20240502 07:59:43.171844 1277 raft_consensus.cc:2781] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:43.171942 1277 raft_consensus.cc:482] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:43.171972 1277 raft_consensus.cc:3037] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.173103 1277 raft_consensus.cc:504] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.173511 1277 leader_election.cc:288] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [CANDIDATE]: Term 1 election: Requested vote from peers 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761), 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:43.173931 1218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6edf17550e3f4a0b945477edc6de6af1" candidate_uuid: "b1e9945160714b16b2db91f90c5ccc70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
I20240502 07:59:43.173996 1067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6edf17550e3f4a0b945477edc6de6af1" candidate_uuid: "b1e9945160714b16b2db91f90c5ccc70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3"
I20240502 07:59:43.174084 1218 raft_consensus.cc:3037] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.174093 1067 raft_consensus.cc:3037] T 6edf17550e3f4a0b945477edc6de6af1 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.175123 1067 raft_consensus.cc:2454] T 6edf17550e3f4a0b945477edc6de6af1 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1e9945160714b16b2db91f90c5ccc70 in term 1.
I20240502 07:59:43.175316 1218 raft_consensus.cc:2454] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1e9945160714b16b2db91f90c5ccc70 in term 1.
I20240502 07:59:43.175570 1107 leader_election.cc:302] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e648af9dbb46fc881340efe1d1e3c3, b1e9945160714b16b2db91f90c5ccc70; no voters:
I20240502 07:59:43.175659 1277 raft_consensus.cc:482] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:43.175750 1277 raft_consensus.cc:504] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.175853 1272 raft_consensus.cc:2781] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:43.176156 1272 raft_consensus.cc:686] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Becoming Leader. State: Replica: b1e9945160714b16b2db91f90c5ccc70, State: Running, Role: LEADER
I20240502 07:59:43.176308 1277 leader_election.cc:288] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761), 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:43.176352 1272 consensus_queue.cc:237] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.176944 1067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682e26b333104f82863b92c545cfdafe" candidate_uuid: "b1e9945160714b16b2db91f90c5ccc70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3" is_pre_election: true
I20240502 07:59:43.177059 1067 raft_consensus.cc:2454] T 682e26b333104f82863b92c545cfdafe P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1e9945160714b16b2db91f90c5ccc70 in term 0.
I20240502 07:59:43.177125 1218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682e26b333104f82863b92c545cfdafe" candidate_uuid: "b1e9945160714b16b2db91f90c5ccc70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" is_pre_election: true
I20240502 07:59:43.177235 1218 raft_consensus.cc:2454] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1e9945160714b16b2db91f90c5ccc70 in term 0.
I20240502 07:59:43.177306 1107 leader_election.cc:302] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e648af9dbb46fc881340efe1d1e3c3, b1e9945160714b16b2db91f90c5ccc70; no voters:
I20240502 07:59:43.177650 1272 raft_consensus.cc:2781] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:43.177717 1272 raft_consensus.cc:482] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:43.177764 1272 raft_consensus.cc:3037] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.178170 963 catalog_manager.cc:5484] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130). New cstate: current_term: 1 leader_uuid: "b1e9945160714b16b2db91f90c5ccc70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:43.178555 1272 raft_consensus.cc:504] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.178915 1272 leader_election.cc:288] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [CANDIDATE]: Term 1 election: Requested vote from peers 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761), 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:43.179344 1218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682e26b333104f82863b92c545cfdafe" candidate_uuid: "b1e9945160714b16b2db91f90c5ccc70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
I20240502 07:59:43.179455 1218 raft_consensus.cc:3037] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.179693 1067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682e26b333104f82863b92c545cfdafe" candidate_uuid: "b1e9945160714b16b2db91f90c5ccc70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3"
I20240502 07:59:43.179805 1067 raft_consensus.cc:3037] T 682e26b333104f82863b92c545cfdafe P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.180256 1218 raft_consensus.cc:2454] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1e9945160714b16b2db91f90c5ccc70 in term 1.
I20240502 07:59:43.180599 1109 leader_election.cc:302] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ec1b26c2f64cc682ed7ae12e133a9f, b1e9945160714b16b2db91f90c5ccc70; no voters:
I20240502 07:59:43.180748 1272 raft_consensus.cc:2781] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:43.180824 1272 raft_consensus.cc:686] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Becoming Leader. State: Replica: b1e9945160714b16b2db91f90c5ccc70, State: Running, Role: LEADER
I20240502 07:59:43.180914 1067 raft_consensus.cc:2454] T 682e26b333104f82863b92c545cfdafe P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1e9945160714b16b2db91f90c5ccc70 in term 1.
I20240502 07:59:43.180935 1272 consensus_queue.cc:237] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.182567 963 catalog_manager.cc:5484] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130). New cstate: current_term: 1 leader_uuid: "b1e9945160714b16b2db91f90c5ccc70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:43.185652 1272 raft_consensus.cc:482] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:43.185750 1272 raft_consensus.cc:504] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.186131 1272 leader_election.cc:288] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761), 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:43.186475 1269 raft_consensus.cc:482] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:43.186522 1218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecebe8dad3946bd956afacf695fabaa" candidate_uuid: "b1e9945160714b16b2db91f90c5ccc70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" is_pre_election: true
I20240502 07:59:43.186643 1067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecebe8dad3946bd956afacf695fabaa" candidate_uuid: "b1e9945160714b16b2db91f90c5ccc70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3" is_pre_election: true
I20240502 07:59:43.186571 1269 raft_consensus.cc:504] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.186702 1218 raft_consensus.cc:2454] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1e9945160714b16b2db91f90c5ccc70 in term 0.
I20240502 07:59:43.186765 1067 raft_consensus.cc:2454] T 2ecebe8dad3946bd956afacf695fabaa P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1e9945160714b16b2db91f90c5ccc70 in term 0.
I20240502 07:59:43.187006 1107 leader_election.cc:302] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e648af9dbb46fc881340efe1d1e3c3, b1e9945160714b16b2db91f90c5ccc70; no voters:
I20240502 07:59:43.187070 1269 leader_election.cc:288] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203), 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:43.187433 1272 raft_consensus.cc:2781] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:43.187506 1272 raft_consensus.cc:482] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:43.187438 1143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a9ebd0c390436b8ace08089f719358" candidate_uuid: "77e648af9dbb46fc881340efe1d1e3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e9945160714b16b2db91f90c5ccc70" is_pre_election: true
I20240502 07:59:43.187438 1218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a9ebd0c390436b8ace08089f719358" candidate_uuid: "77e648af9dbb46fc881340efe1d1e3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" is_pre_election: true
I20240502 07:59:43.187554 1272 raft_consensus.cc:3037] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.187611 1143 raft_consensus.cc:2454] T 69a9ebd0c390436b8ace08089f719358 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e648af9dbb46fc881340efe1d1e3c3 in term 0.
I20240502 07:59:43.187626 1218 raft_consensus.cc:2454] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e648af9dbb46fc881340efe1d1e3c3 in term 0.
I20240502 07:59:43.187901 1033 leader_election.cc:302] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ec1b26c2f64cc682ed7ae12e133a9f, 77e648af9dbb46fc881340efe1d1e3c3; no voters:
I20240502 07:59:43.188092 1269 raft_consensus.cc:2781] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:43.188150 1269 raft_consensus.cc:482] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:43.188185 1269 raft_consensus.cc:3037] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.188714 1272 raft_consensus.cc:504] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.189131 1272 leader_election.cc:288] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [CANDIDATE]: Term 1 election: Requested vote from peers 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761), 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:43.189102 1269 raft_consensus.cc:504] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.189545 1269 leader_election.cc:288] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [CANDIDATE]: Term 1 election: Requested vote from peers b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203), 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:43.189741 1143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a9ebd0c390436b8ace08089f719358" candidate_uuid: "77e648af9dbb46fc881340efe1d1e3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
I20240502 07:59:43.189934 1143 raft_consensus.cc:3037] T 69a9ebd0c390436b8ace08089f719358 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.190814 1143 raft_consensus.cc:2454] T 69a9ebd0c390436b8ace08089f719358 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e648af9dbb46fc881340efe1d1e3c3 in term 1.
I20240502 07:59:43.189698 1218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecebe8dad3946bd956afacf695fabaa" candidate_uuid: "b1e9945160714b16b2db91f90c5ccc70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
I20240502 07:59:43.191016 1218 raft_consensus.cc:3037] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.191183 1030 leader_election.cc:302] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e648af9dbb46fc881340efe1d1e3c3, b1e9945160714b16b2db91f90c5ccc70; no voters:
I20240502 07:59:43.191356 1269 raft_consensus.cc:2781] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:43.191438 1269 raft_consensus.cc:686] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Becoming Leader. State: Replica: 77e648af9dbb46fc881340efe1d1e3c3, State: Running, Role: LEADER
I20240502 07:59:43.191596 1269 consensus_queue.cc:237] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.191890 1218 raft_consensus.cc:2454] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1e9945160714b16b2db91f90c5ccc70 in term 1.
I20240502 07:59:43.189797 1067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecebe8dad3946bd956afacf695fabaa" candidate_uuid: "b1e9945160714b16b2db91f90c5ccc70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3"
I20240502 07:59:43.189698 1217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a9ebd0c390436b8ace08089f719358" candidate_uuid: "77e648af9dbb46fc881340efe1d1e3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
I20240502 07:59:43.192124 1217 raft_consensus.cc:3037] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.192132 1109 leader_election.cc:302] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ec1b26c2f64cc682ed7ae12e133a9f, b1e9945160714b16b2db91f90c5ccc70; no voters:
I20240502 07:59:43.192359 1067 raft_consensus.cc:3037] T 2ecebe8dad3946bd956afacf695fabaa P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.193035 1272 raft_consensus.cc:2781] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:43.193066 1067 raft_consensus.cc:2454] T 2ecebe8dad3946bd956afacf695fabaa P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1e9945160714b16b2db91f90c5ccc70 in term 1.
I20240502 07:59:43.193363 1217 raft_consensus.cc:2454] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e648af9dbb46fc881340efe1d1e3c3 in term 1.
I20240502 07:59:43.193661 1272 raft_consensus.cc:686] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Becoming Leader. State: Replica: b1e9945160714b16b2db91f90c5ccc70, State: Running, Role: LEADER
I20240502 07:59:43.193590 963 catalog_manager.cc:5484] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129). New cstate: current_term: 1 leader_uuid: "77e648af9dbb46fc881340efe1d1e3c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:43.193804 1272 consensus_queue.cc:237] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.195662 964 catalog_manager.cc:5484] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130). New cstate: current_term: 1 leader_uuid: "b1e9945160714b16b2db91f90c5ccc70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:43.225901 1269 raft_consensus.cc:482] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:43.226128 1269 raft_consensus.cc:504] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.226751 1269 leader_election.cc:288] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203), 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:43.227337 1217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7c1536618ae4e74aa79349d04201fb9" candidate_uuid: "77e648af9dbb46fc881340efe1d1e3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" is_pre_election: true
I20240502 07:59:43.227336 1143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7c1536618ae4e74aa79349d04201fb9" candidate_uuid: "77e648af9dbb46fc881340efe1d1e3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e9945160714b16b2db91f90c5ccc70" is_pre_election: true
I20240502 07:59:43.227556 1217 raft_consensus.cc:2454] T c7c1536618ae4e74aa79349d04201fb9 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e648af9dbb46fc881340efe1d1e3c3 in term 0.
I20240502 07:59:43.227581 1143 raft_consensus.cc:2454] T c7c1536618ae4e74aa79349d04201fb9 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e648af9dbb46fc881340efe1d1e3c3 in term 0.
I20240502 07:59:43.227932 1030 leader_election.cc:302] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e648af9dbb46fc881340efe1d1e3c3, b1e9945160714b16b2db91f90c5ccc70; no voters:
I20240502 07:59:43.228133 1269 raft_consensus.cc:2781] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:43.228192 1269 raft_consensus.cc:482] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:43.228236 1269 raft_consensus.cc:3037] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.229252 1269 raft_consensus.cc:504] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.229774 1269 leader_election.cc:288] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [CANDIDATE]: Term 1 election: Requested vote from peers b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203), 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:43.230221 1217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7c1536618ae4e74aa79349d04201fb9" candidate_uuid: "77e648af9dbb46fc881340efe1d1e3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
I20240502 07:59:43.230260 1143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7c1536618ae4e74aa79349d04201fb9" candidate_uuid: "77e648af9dbb46fc881340efe1d1e3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
I20240502 07:59:43.230382 1217 raft_consensus.cc:3037] T c7c1536618ae4e74aa79349d04201fb9 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.230384 1143 raft_consensus.cc:3037] T c7c1536618ae4e74aa79349d04201fb9 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.231289 1143 raft_consensus.cc:2454] T c7c1536618ae4e74aa79349d04201fb9 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e648af9dbb46fc881340efe1d1e3c3 in term 1.
I20240502 07:59:43.231325 1217 raft_consensus.cc:2454] T c7c1536618ae4e74aa79349d04201fb9 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e648af9dbb46fc881340efe1d1e3c3 in term 1.
I20240502 07:59:43.231686 1030 leader_election.cc:302] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e648af9dbb46fc881340efe1d1e3c3, b1e9945160714b16b2db91f90c5ccc70; no voters:
I20240502 07:59:43.231902 1269 raft_consensus.cc:2781] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:43.232002 1269 raft_consensus.cc:686] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Becoming Leader. State: Replica: 77e648af9dbb46fc881340efe1d1e3c3, State: Running, Role: LEADER
I20240502 07:59:43.232142 1269 consensus_queue.cc:237] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.234112 964 catalog_manager.cc:5484] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129). New cstate: current_term: 1 leader_uuid: "77e648af9dbb46fc881340efe1d1e3c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:43.238754 1276 raft_consensus.cc:482] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:43.238893 1276 raft_consensus.cc:504] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.239480 1276 leader_election.cc:288] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761), b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203)
I20240502 07:59:43.243618 1272 raft_consensus.cc:482] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:43.243728 1272 raft_consensus.cc:504] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.244107 1272 leader_election.cc:288] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761), 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:43.244503 1067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1b6523d6724a02acc9ae19f3f0255e" candidate_uuid: "b1e9945160714b16b2db91f90c5ccc70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3" is_pre_election: true
I20240502 07:59:43.244534 1217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1b6523d6724a02acc9ae19f3f0255e" candidate_uuid: "b1e9945160714b16b2db91f90c5ccc70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" is_pre_election: true
I20240502 07:59:43.244680 1067 raft_consensus.cc:2454] T 2c1b6523d6724a02acc9ae19f3f0255e P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1e9945160714b16b2db91f90c5ccc70 in term 0.
I20240502 07:59:43.244693 1217 raft_consensus.cc:2454] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1e9945160714b16b2db91f90c5ccc70 in term 0.
I20240502 07:59:43.244922 1143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1b6523d6724a02acc9ae19f3f0255e" candidate_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e9945160714b16b2db91f90c5ccc70" is_pre_election: true
I20240502 07:59:43.244993 1107 leader_election.cc:302] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e648af9dbb46fc881340efe1d1e3c3, b1e9945160714b16b2db91f90c5ccc70; no voters:
I20240502 07:59:43.245051 1143 raft_consensus.cc:2454] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27ec1b26c2f64cc682ed7ae12e133a9f in term 0.
I20240502 07:59:43.245167 1272 raft_consensus.cc:2781] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:43.245230 1272 raft_consensus.cc:482] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:43.245285 1272 raft_consensus.cc:3037] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.245394 1181 leader_election.cc:302] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ec1b26c2f64cc682ed7ae12e133a9f, b1e9945160714b16b2db91f90c5ccc70; no voters:
I20240502 07:59:43.245556 1276 raft_consensus.cc:2781] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:43.245613 1276 raft_consensus.cc:482] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:43.245647 1276 raft_consensus.cc:3037] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.245667 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1b6523d6724a02acc9ae19f3f0255e" candidate_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3" is_pre_election: true
I20240502 07:59:43.245749 1066 raft_consensus.cc:2454] T 2c1b6523d6724a02acc9ae19f3f0255e P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27ec1b26c2f64cc682ed7ae12e133a9f in term 0.
I20240502 07:59:43.246412 1272 raft_consensus.cc:504] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.246522 1276 raft_consensus.cc:504] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.246819 1276 leader_election.cc:288] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [CANDIDATE]: Term 1 election: Requested vote from peers 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761), b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203)
I20240502 07:59:43.246819 1272 leader_election.cc:288] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [CANDIDATE]: Term 1 election: Requested vote from peers 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761), 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:43.247109 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1b6523d6724a02acc9ae19f3f0255e" candidate_uuid: "b1e9945160714b16b2db91f90c5ccc70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3"
I20240502 07:59:43.247192 1143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1b6523d6724a02acc9ae19f3f0255e" candidate_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
I20240502 07:59:43.247233 1066 raft_consensus.cc:3037] T 2c1b6523d6724a02acc9ae19f3f0255e P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.247341 1143 raft_consensus.cc:2379] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 27ec1b26c2f64cc682ed7ae12e133a9f in current term 1: Already voted for candidate b1e9945160714b16b2db91f90c5ccc70 in this term.
I20240502 07:59:43.248256 1066 raft_consensus.cc:2454] T 2c1b6523d6724a02acc9ae19f3f0255e P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1e9945160714b16b2db91f90c5ccc70 in term 1.
I20240502 07:59:43.248584 1107 leader_election.cc:302] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e648af9dbb46fc881340efe1d1e3c3, b1e9945160714b16b2db91f90c5ccc70; no voters:
I20240502 07:59:43.248715 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1b6523d6724a02acc9ae19f3f0255e" candidate_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3"
I20240502 07:59:43.248805 1217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1b6523d6724a02acc9ae19f3f0255e" candidate_uuid: "b1e9945160714b16b2db91f90c5ccc70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
I20240502 07:59:43.248845 1066 raft_consensus.cc:2379] T 2c1b6523d6724a02acc9ae19f3f0255e P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 27ec1b26c2f64cc682ed7ae12e133a9f in current term 1: Already voted for candidate b1e9945160714b16b2db91f90c5ccc70 in this term.
I20240502 07:59:43.248952 1217 raft_consensus.cc:2379] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b1e9945160714b16b2db91f90c5ccc70 in current term 1: Already voted for candidate 27ec1b26c2f64cc682ed7ae12e133a9f in this term.
I20240502 07:59:43.249125 1272 raft_consensus.cc:2781] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:43.249217 1272 raft_consensus.cc:686] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Becoming Leader. State: Replica: b1e9945160714b16b2db91f90c5ccc70, State: Running, Role: LEADER
I20240502 07:59:43.249264 1182 leader_election.cc:302] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27ec1b26c2f64cc682ed7ae12e133a9f; no voters: 77e648af9dbb46fc881340efe1d1e3c3, b1e9945160714b16b2db91f90c5ccc70
I20240502 07:59:43.249373 1272 consensus_queue.cc:237] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.249430 1276 raft_consensus.cc:2726] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 07:59:43.251160 964 catalog_manager.cc:5484] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130). New cstate: current_term: 1 leader_uuid: "b1e9945160714b16b2db91f90c5ccc70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:43.254704 1314 raft_consensus.cc:482] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:43.254838 1314 raft_consensus.cc:504] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.255267 1314 leader_election.cc:288] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761), 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:43.255661 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf0ff2967c040a589f915172d3f446b" candidate_uuid: "b1e9945160714b16b2db91f90c5ccc70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3" is_pre_election: true
I20240502 07:59:43.255798 1066 raft_consensus.cc:2454] T cdf0ff2967c040a589f915172d3f446b P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1e9945160714b16b2db91f90c5ccc70 in term 0.
I20240502 07:59:43.255811 1217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf0ff2967c040a589f915172d3f446b" candidate_uuid: "b1e9945160714b16b2db91f90c5ccc70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" is_pre_election: true
I20240502 07:59:43.256002 1217 raft_consensus.cc:2454] T cdf0ff2967c040a589f915172d3f446b P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1e9945160714b16b2db91f90c5ccc70 in term 0.
I20240502 07:59:43.256085 1107 leader_election.cc:302] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e648af9dbb46fc881340efe1d1e3c3, b1e9945160714b16b2db91f90c5ccc70; no voters:
I20240502 07:59:43.256289 1314 raft_consensus.cc:2781] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:43.256356 1314 raft_consensus.cc:482] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:43.256397 1314 raft_consensus.cc:3037] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.257347 1269 raft_consensus.cc:482] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:43.257438 1314 raft_consensus.cc:504] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.257440 1269 raft_consensus.cc:504] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.257866 1269 leader_election.cc:288] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203), 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:43.258203 1217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4233b549b34b01b6f9662c7e7b3515" candidate_uuid: "77e648af9dbb46fc881340efe1d1e3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" is_pre_election: true
I20240502 07:59:43.258273 1143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4233b549b34b01b6f9662c7e7b3515" candidate_uuid: "77e648af9dbb46fc881340efe1d1e3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e9945160714b16b2db91f90c5ccc70" is_pre_election: true
I20240502 07:59:43.257866 1314 leader_election.cc:288] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [CANDIDATE]: Term 1 election: Requested vote from peers 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761), 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:43.258344 1217 raft_consensus.cc:2454] T df4233b549b34b01b6f9662c7e7b3515 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e648af9dbb46fc881340efe1d1e3c3 in term 0.
I20240502 07:59:43.258334 1218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf0ff2967c040a589f915172d3f446b" candidate_uuid: "b1e9945160714b16b2db91f90c5ccc70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
I20240502 07:59:43.258334 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf0ff2967c040a589f915172d3f446b" candidate_uuid: "b1e9945160714b16b2db91f90c5ccc70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3"
I20240502 07:59:43.258352 1143 raft_consensus.cc:2454] T df4233b549b34b01b6f9662c7e7b3515 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e648af9dbb46fc881340efe1d1e3c3 in term 0.
I20240502 07:59:43.258518 1066 raft_consensus.cc:3037] T cdf0ff2967c040a589f915172d3f446b P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.258525 1218 raft_consensus.cc:3037] T cdf0ff2967c040a589f915172d3f446b P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.258716 1033 leader_election.cc:302] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ec1b26c2f64cc682ed7ae12e133a9f, 77e648af9dbb46fc881340efe1d1e3c3; no voters:
I20240502 07:59:43.258929 1269 raft_consensus.cc:2781] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:43.258987 1269 raft_consensus.cc:482] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:43.259022 1269 raft_consensus.cc:3037] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.259625 1218 raft_consensus.cc:2454] T cdf0ff2967c040a589f915172d3f446b P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1e9945160714b16b2db91f90c5ccc70 in term 1.
I20240502 07:59:43.259630 1066 raft_consensus.cc:2454] T cdf0ff2967c040a589f915172d3f446b P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1e9945160714b16b2db91f90c5ccc70 in term 1.
I20240502 07:59:43.259938 1269 raft_consensus.cc:504] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.259980 1109 leader_election.cc:302] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ec1b26c2f64cc682ed7ae12e133a9f, b1e9945160714b16b2db91f90c5ccc70; no voters:
I20240502 07:59:43.260143 1314 raft_consensus.cc:2781] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:43.260233 1314 raft_consensus.cc:686] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Becoming Leader. State: Replica: b1e9945160714b16b2db91f90c5ccc70, State: Running, Role: LEADER
I20240502 07:59:43.260334 1269 leader_election.cc:288] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [CANDIDATE]: Term 1 election: Requested vote from peers b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203), 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:43.260480 1314 consensus_queue.cc:237] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.260619 1143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4233b549b34b01b6f9662c7e7b3515" candidate_uuid: "77e648af9dbb46fc881340efe1d1e3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
I20240502 07:59:43.260671 1218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4233b549b34b01b6f9662c7e7b3515" candidate_uuid: "77e648af9dbb46fc881340efe1d1e3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
I20240502 07:59:43.260700 1143 raft_consensus.cc:3037] T df4233b549b34b01b6f9662c7e7b3515 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.260751 1218 raft_consensus.cc:3037] T df4233b549b34b01b6f9662c7e7b3515 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.261408 1218 raft_consensus.cc:2454] T df4233b549b34b01b6f9662c7e7b3515 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e648af9dbb46fc881340efe1d1e3c3 in term 1.
I20240502 07:59:43.261718 1143 raft_consensus.cc:2454] T df4233b549b34b01b6f9662c7e7b3515 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e648af9dbb46fc881340efe1d1e3c3 in term 1.
I20240502 07:59:43.261723 1033 leader_election.cc:302] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ec1b26c2f64cc682ed7ae12e133a9f, 77e648af9dbb46fc881340efe1d1e3c3; no voters:
I20240502 07:59:43.261940 1269 raft_consensus.cc:2781] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:43.262115 1269 raft_consensus.cc:686] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Becoming Leader. State: Replica: 77e648af9dbb46fc881340efe1d1e3c3, State: Running, Role: LEADER
I20240502 07:59:43.262261 1269 consensus_queue.cc:237] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.262544 964 catalog_manager.cc:5484] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130). New cstate: current_term: 1 leader_uuid: "b1e9945160714b16b2db91f90c5ccc70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:43.264045 963 catalog_manager.cc:5484] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129). New cstate: current_term: 1 leader_uuid: "77e648af9dbb46fc881340efe1d1e3c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:43.318431 1276 raft_consensus.cc:482] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:43.318598 1276 raft_consensus.cc:504] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.319003 1276 leader_election.cc:288] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761), b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203)
I20240502 07:59:43.319453 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c5f52a420c467684b2cb50972e7c99" candidate_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3" is_pre_election: true
I20240502 07:59:43.319453 1143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c5f52a420c467684b2cb50972e7c99" candidate_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e9945160714b16b2db91f90c5ccc70" is_pre_election: true
I20240502 07:59:43.319617 1143 raft_consensus.cc:2454] T 58c5f52a420c467684b2cb50972e7c99 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27ec1b26c2f64cc682ed7ae12e133a9f in term 0.
I20240502 07:59:43.319617 1066 raft_consensus.cc:2454] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27ec1b26c2f64cc682ed7ae12e133a9f in term 0.
I20240502 07:59:43.319983 1182 leader_election.cc:302] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ec1b26c2f64cc682ed7ae12e133a9f, 77e648af9dbb46fc881340efe1d1e3c3; no voters:
I20240502 07:59:43.320196 1276 raft_consensus.cc:2781] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:43.320282 1276 raft_consensus.cc:482] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:43.320315 1276 raft_consensus.cc:3037] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.321127 1276 raft_consensus.cc:504] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.321484 1276 leader_election.cc:288] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [CANDIDATE]: Term 1 election: Requested vote from peers 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761), b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203)
I20240502 07:59:43.321898 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c5f52a420c467684b2cb50972e7c99" candidate_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3"
I20240502 07:59:43.321928 1143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c5f52a420c467684b2cb50972e7c99" candidate_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
I20240502 07:59:43.321993 1066 raft_consensus.cc:3037] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.322058 1143 raft_consensus.cc:3037] T 58c5f52a420c467684b2cb50972e7c99 P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.322676 1066 raft_consensus.cc:2454] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27ec1b26c2f64cc682ed7ae12e133a9f in term 1.
I20240502 07:59:43.322813 1143 raft_consensus.cc:2454] T 58c5f52a420c467684b2cb50972e7c99 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27ec1b26c2f64cc682ed7ae12e133a9f in term 1.
I20240502 07:59:43.322983 1182 leader_election.cc:302] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ec1b26c2f64cc682ed7ae12e133a9f, 77e648af9dbb46fc881340efe1d1e3c3; no voters:
I20240502 07:59:43.323199 1276 raft_consensus.cc:2781] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:43.323424 1276 raft_consensus.cc:686] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 LEADER]: Becoming Leader. State: Replica: 27ec1b26c2f64cc682ed7ae12e133a9f, State: Running, Role: LEADER
I20240502 07:59:43.323575 1276 consensus_queue.cc:237] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.325107 963 catalog_manager.cc:5484] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f reported cstate change: term changed from 0 to 1, leader changed from <none> to 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131). New cstate: current_term: 1 leader_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: HEALTHY } } }
I20240502 07:59:43.392729 1269 raft_consensus.cc:482] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:43.392884 1269 raft_consensus.cc:504] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.393262 1269 leader_election.cc:288] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203), 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:43.393716 1218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d982dd52d246348fb3984fd1c3a40f" candidate_uuid: "77e648af9dbb46fc881340efe1d1e3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" is_pre_election: true
I20240502 07:59:43.393716 1143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d982dd52d246348fb3984fd1c3a40f" candidate_uuid: "77e648af9dbb46fc881340efe1d1e3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e9945160714b16b2db91f90c5ccc70" is_pre_election: true
I20240502 07:59:43.393873 1218 raft_consensus.cc:2454] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e648af9dbb46fc881340efe1d1e3c3 in term 0.
I20240502 07:59:43.393891 1143 raft_consensus.cc:2454] T d0d982dd52d246348fb3984fd1c3a40f P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e648af9dbb46fc881340efe1d1e3c3 in term 0.
I20240502 07:59:43.394156 1030 leader_election.cc:302] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e648af9dbb46fc881340efe1d1e3c3, b1e9945160714b16b2db91f90c5ccc70; no voters:
I20240502 07:59:43.394353 1269 raft_consensus.cc:2781] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:43.394414 1269 raft_consensus.cc:482] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:43.394455 1269 raft_consensus.cc:3037] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.395249 1269 raft_consensus.cc:504] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.395557 1269 leader_election.cc:288] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [CANDIDATE]: Term 1 election: Requested vote from peers b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203), 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:43.395884 1143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d982dd52d246348fb3984fd1c3a40f" candidate_uuid: "77e648af9dbb46fc881340efe1d1e3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
I20240502 07:59:43.395934 1218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d982dd52d246348fb3984fd1c3a40f" candidate_uuid: "77e648af9dbb46fc881340efe1d1e3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
I20240502 07:59:43.395989 1143 raft_consensus.cc:3037] T d0d982dd52d246348fb3984fd1c3a40f P b1e9945160714b16b2db91f90c5ccc70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.396019 1218 raft_consensus.cc:3037] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.396749 1218 raft_consensus.cc:2454] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e648af9dbb46fc881340efe1d1e3c3 in term 1.
I20240502 07:59:43.396764 1143 raft_consensus.cc:2454] T d0d982dd52d246348fb3984fd1c3a40f P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e648af9dbb46fc881340efe1d1e3c3 in term 1.
I20240502 07:59:43.396983 1033 leader_election.cc:302] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ec1b26c2f64cc682ed7ae12e133a9f, 77e648af9dbb46fc881340efe1d1e3c3; no voters:
I20240502 07:59:43.397132 1269 raft_consensus.cc:2781] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:43.397209 1269 raft_consensus.cc:686] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Becoming Leader. State: Replica: 77e648af9dbb46fc881340efe1d1e3c3, State: Running, Role: LEADER
I20240502 07:59:43.397320 1269 consensus_queue.cc:237] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.398939 963 catalog_manager.cc:5484] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129). New cstate: current_term: 1 leader_uuid: "77e648af9dbb46fc881340efe1d1e3c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:43.440254 1218 raft_consensus.cc:1261] T cdf0ff2967c040a589f915172d3f446b P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:43.440804 1314 consensus_queue.cc:1035] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:43.441349 1217 raft_consensus.cc:3037] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.441493 1218 raft_consensus.cc:1261] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:43.441891 1314 consensus_queue.cc:1035] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:43.442463 1217 raft_consensus.cc:1261] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:43.442775 1066 raft_consensus.cc:1261] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:43.443307 1143 raft_consensus.cc:1261] T 58c5f52a420c467684b2cb50972e7c99 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:43.443400 1276 consensus_queue.cc:1035] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:43.443591 1314 consensus_queue.cc:1035] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:43.443603 1215 raft_consensus.cc:1261] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:43.444718 1325 consensus_queue.cc:1035] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:43.445124 1272 consensus_queue.cc:1035] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:43.445748 1065 raft_consensus.cc:1261] T 682e26b333104f82863b92c545cfdafe P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:43.445837 1067 raft_consensus.cc:1261] T cdf0ff2967c040a589f915172d3f446b P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:43.446110 1064 raft_consensus.cc:1261] T 5437f94679a841e3980053e69d624d76 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:43.446403 1063 raft_consensus.cc:1261] T 2ecebe8dad3946bd956afacf695fabaa P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:43.448995 1314 consensus_queue.cc:1035] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:43.449126 1063 raft_consensus.cc:1261] T 2c1b6523d6724a02acc9ae19f3f0255e P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:43.449270 1314 consensus_queue.cc:1035] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:43.449433 1277 consensus_queue.cc:1035] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:43.449505 1314 consensus_queue.cc:1035] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:43.448997 1064 raft_consensus.cc:1261] T 6edf17550e3f4a0b945477edc6de6af1 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:43.450013 1214 raft_consensus.cc:1261] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:43.450282 1213 raft_consensus.cc:1261] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:43.450805 1272 consensus_queue.cc:1035] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:43.450904 1314 consensus_queue.cc:1035] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:43.452674 1272 consensus_queue.cc:1035] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:43.452948 1272 consensus_queue.cc:1035] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:43.465898 1211 raft_consensus.cc:1261] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:43.466238 1210 raft_consensus.cc:1261] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:43.466523 1209 raft_consensus.cc:1261] T c7c1536618ae4e74aa79349d04201fb9 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:43.466879 1218 raft_consensus.cc:3037] T fda98cfe14f947969fb41e26c75bc384 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:43.467937 1218 raft_consensus.cc:1261] T fda98cfe14f947969fb41e26c75bc384 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:43.472144 1311 consensus_queue.cc:1035] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:43.472543 1311 consensus_queue.cc:1035] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:43.472795 1311 consensus_queue.cc:1035] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:43.473043 1311 consensus_queue.cc:1035] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:43.474946 1143 raft_consensus.cc:1261] T fda98cfe14f947969fb41e26c75bc384 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:43.475189 1142 raft_consensus.cc:1261] T 69a9ebd0c390436b8ace08089f719358 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:43.475474 1141 raft_consensus.cc:1261] T d0d982dd52d246348fb3984fd1c3a40f P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:43.475621 1140 raft_consensus.cc:1261] T c7c1536618ae4e74aa79349d04201fb9 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:43.475457 1379 consensus_queue.cc:1035] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:43.475922 1139 raft_consensus.cc:1261] T df4233b549b34b01b6f9662c7e7b3515 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 07:59:43.476491 1379 consensus_queue.cc:1035] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:43.477425 1269 consensus_queue.cc:1035] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 07:59:43.478446 1361 mvcc.cc:205] Tried to move back new op lower bound from 7023152265003331584 to 7023152263597211648. Current Snapshot: MvccSnapshot[applied={T|T < 7023152264972713984}]
I20240502 07:59:43.488529 1209 raft_consensus.cc:1261] T df4233b549b34b01b6f9662c7e7b3515 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 07:59:43.492568 1380 consensus_queue.cc:1035] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:43.501155 1379 consensus_queue.cc:1035] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:43.501526 1379 consensus_queue.cc:1035] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:43.507414 1356 mvcc.cc:205] Tried to move back new op lower bound from 7023152264967323648 to 7023152264236756992. Current Snapshot: MvccSnapshot[applied={T|T < 7023152264967323648}]
I20240502 07:59:43.576416 1335 mvcc.cc:205] Tried to move back new op lower bound from 7023152265109438464 to 7023152264243621888. Current Snapshot: MvccSnapshot[applied={T|T < 7023152265069154304}]
I20240502 07:59:43.804584 1018 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:59:43.879943 1143 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2ecebe8dad3946bd956afacf695fabaa"
dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
mode: GRACEFUL
new_leader_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
from {username='slave'} at 127.0.0.1:47914
I20240502 07:59:43.880115 1143 raft_consensus.cc:595] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Received request to transfer leadership to TS 27ec1b26c2f64cc682ed7ae12e133a9f
I20240502 07:59:43.880878 1141 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "682e26b333104f82863b92c545cfdafe"
dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
mode: GRACEFUL
new_leader_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
from {username='slave'} at 127.0.0.1:47914
I20240502 07:59:43.880996 1141 raft_consensus.cc:595] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Received request to transfer leadership to TS 27ec1b26c2f64cc682ed7ae12e133a9f
I20240502 07:59:43.901109 1381 raft_consensus.cc:979] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70: : Instructing follower 27ec1b26c2f64cc682ed7ae12e133a9f to start an election
I20240502 07:59:43.901243 1381 raft_consensus.cc:1067] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Signalling peer 27ec1b26c2f64cc682ed7ae12e133a9f to start an election
W20240502 07:59:43.910604 1123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47900: Service unavailable: leader transfer in progress
I20240502 07:59:43.915689 1064 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "69a9ebd0c390436b8ace08089f719358"
dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3"
mode: GRACEFUL
new_leader_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
from {username='slave'} at 127.0.0.1:48332
I20240502 07:59:43.915858 1064 raft_consensus.cc:595] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Received request to transfer leadership to TS 27ec1b26c2f64cc682ed7ae12e133a9f
I20240502 07:59:43.916134 1018 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240502 07:59:43.916289 1018 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:59:43.917121 1381 raft_consensus.cc:979] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70: : Instructing follower 27ec1b26c2f64cc682ed7ae12e133a9f to start an election
I20240502 07:59:43.917232 1385 raft_consensus.cc:1067] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Signalling peer 27ec1b26c2f64cc682ed7ae12e133a9f to start an election
I20240502 07:59:43.920305 1376 raft_consensus.cc:979] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3: : Instructing follower 27ec1b26c2f64cc682ed7ae12e133a9f to start an election
I20240502 07:59:43.920389 1376 raft_consensus.cc:1067] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Signalling peer 27ec1b26c2f64cc682ed7ae12e133a9f to start an election
I20240502 07:59:43.921058 1212 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "69a9ebd0c390436b8ace08089f719358"
dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
from {username='slave'} at 127.0.0.1:52060
I20240502 07:59:43.921240 1212 raft_consensus.cc:482] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:43.921365 1212 raft_consensus.cc:3037] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:43.922559 1215 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "682e26b333104f82863b92c545cfdafe"
dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
from {username='slave'} at 127.0.0.1:52044
I20240502 07:59:43.922672 1215 raft_consensus.cc:482] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:43.922725 1215 raft_consensus.cc:3037] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:43.923751 1215 raft_consensus.cc:504] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.924108 1215 leader_election.cc:288] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [CANDIDATE]: Term 2 election: Requested vote from peers 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761), b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203)
I20240502 07:59:43.924430 1141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682e26b333104f82863b92c545cfdafe" candidate_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: true dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
I20240502 07:59:43.924551 1141 raft_consensus.cc:3032] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:43.924592 1141 raft_consensus.cc:729] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b1e9945160714b16b2db91f90c5ccc70, State: Running, Role: LEADER
I20240502 07:59:43.924710 1141 consensus_queue.cc:260] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 1.22, Last appended by leader: 22, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.924820 1141 raft_consensus.cc:3037] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:43.926192 1141 raft_consensus.cc:2454] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27ec1b26c2f64cc682ed7ae12e133a9f in term 2.
I20240502 07:59:43.926420 1181 leader_election.cc:302] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ec1b26c2f64cc682ed7ae12e133a9f, b1e9945160714b16b2db91f90c5ccc70; no voters:
I20240502 07:59:43.926841 1400 raft_consensus.cc:2781] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:43.926918 1400 raft_consensus.cc:686] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Becoming Leader. State: Replica: 27ec1b26c2f64cc682ed7ae12e133a9f, State: Running, Role: LEADER
I20240502 07:59:43.927223 1400 consensus_queue.cc:237] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 1.22, Last appended by leader: 22, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
W20240502 07:59:43.927961 1123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47900: Illegal state: replica b1e9945160714b16b2db91f90c5ccc70 is not leader of this config: current role FOLLOWER
W20240502 07:59:43.929525 1123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47900: Service unavailable: leader transfer in progress
W20240502 07:59:43.930792 1123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47900: Illegal state: replica b1e9945160714b16b2db91f90c5ccc70 is not leader of this config: current role FOLLOWER
W20240502 07:59:43.931298 1123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47900: Service unavailable: leader transfer in progress
W20240502 07:59:43.931841 1042 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48328: Service unavailable: leader transfer in progress
W20240502 07:59:43.932492 1042 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48328: Service unavailable: leader transfer in progress
I20240502 07:59:43.933784 1212 raft_consensus.cc:504] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.934233 1212 leader_election.cc:288] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [CANDIDATE]: Term 2 election: Requested vote from peers 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761), b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203)
I20240502 07:59:43.935771 1141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a9ebd0c390436b8ace08089f719358" candidate_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: true dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
I20240502 07:59:43.935946 1141 raft_consensus.cc:3037] T 69a9ebd0c390436b8ace08089f719358 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:43.936117 1218 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2ecebe8dad3946bd956afacf695fabaa"
dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
from {username='slave'} at 127.0.0.1:52044
I20240502 07:59:43.936226 1218 raft_consensus.cc:482] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
W20240502 07:59:43.935832 1123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47900: Illegal state: replica b1e9945160714b16b2db91f90c5ccc70 is not leader of this config: current role FOLLOWER
I20240502 07:59:43.936496 1218 raft_consensus.cc:3037] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:43.937295 1141 raft_consensus.cc:2454] T 69a9ebd0c390436b8ace08089f719358 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27ec1b26c2f64cc682ed7ae12e133a9f in term 2.
I20240502 07:59:43.937597 1181 leader_election.cc:302] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ec1b26c2f64cc682ed7ae12e133a9f, b1e9945160714b16b2db91f90c5ccc70; no voters:
I20240502 07:59:43.937773 1276 raft_consensus.cc:2781] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:43.937840 1276 raft_consensus.cc:686] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Becoming Leader. State: Replica: 27ec1b26c2f64cc682ed7ae12e133a9f, State: Running, Role: LEADER
I20240502 07:59:43.937297 1218 raft_consensus.cc:504] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.937973 1276 consensus_queue.cc:237] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 24, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.938450 1218 leader_election.cc:288] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [CANDIDATE]: Term 2 election: Requested vote from peers b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203), 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761)
I20240502 07:59:43.939397 1143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecebe8dad3946bd956afacf695fabaa" candidate_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: true dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
I20240502 07:59:43.939520 1143 raft_consensus.cc:3032] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:43.939563 1143 raft_consensus.cc:729] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b1e9945160714b16b2db91f90c5ccc70, State: Running, Role: LEADER
I20240502 07:59:43.939711 1143 consensus_queue.cc:260] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 1.22, Last appended by leader: 22, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.939834 1143 raft_consensus.cc:3037] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:43.941192 1143 raft_consensus.cc:2454] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27ec1b26c2f64cc682ed7ae12e133a9f in term 2.
I20240502 07:59:43.942075 1181 leader_election.cc:302] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ec1b26c2f64cc682ed7ae12e133a9f, b1e9945160714b16b2db91f90c5ccc70; no voters:
I20240502 07:59:43.942262 1276 raft_consensus.cc:2781] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:43.942358 1276 raft_consensus.cc:686] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Becoming Leader. State: Replica: 27ec1b26c2f64cc682ed7ae12e133a9f, State: Running, Role: LEADER
I20240502 07:59:43.942394 1064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682e26b333104f82863b92c545cfdafe" candidate_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: true dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3"
I20240502 07:59:43.942517 1064 raft_consensus.cc:3037] T 682e26b333104f82863b92c545cfdafe P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:43.942494 1276 consensus_queue.cc:237] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 1.22, Last appended by leader: 22, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.943554 1064 raft_consensus.cc:2454] T 682e26b333104f82863b92c545cfdafe P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27ec1b26c2f64cc682ed7ae12e133a9f in term 2.
W20240502 07:59:43.944643 1042 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48328: Service unavailable: leader transfer in progress
I20240502 07:59:43.945703 956 catalog_manager.cc:5484] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f reported cstate change: term changed from 1 to 2, leader changed from b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130) to 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131). New cstate: current_term: 2 leader_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: HEALTHY } } }
I20240502 07:59:43.946149 1143 raft_consensus.cc:1226] T 69a9ebd0c390436b8ace08089f719358 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Rejecting Update request from peer 77e648af9dbb46fc881340efe1d1e3c3 for earlier term 1. Current term is 2. Ops: [1.24-1.24]
I20240502 07:59:43.947511 1064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a9ebd0c390436b8ace08089f719358" candidate_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: true dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3"
I20240502 07:59:43.947636 1064 raft_consensus.cc:3032] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:43.947679 1064 raft_consensus.cc:729] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77e648af9dbb46fc881340efe1d1e3c3, State: Running, Role: LEADER
I20240502 07:59:43.947816 1064 consensus_queue.cc:260] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 24, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:43.947985 1064 raft_consensus.cc:3037] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:43.949266 1064 raft_consensus.cc:2454] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27ec1b26c2f64cc682ed7ae12e133a9f in term 2.
I20240502 07:59:43.950294 1064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecebe8dad3946bd956afacf695fabaa" candidate_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: true dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3"
I20240502 07:59:43.950399 1064 raft_consensus.cc:3037] T 2ecebe8dad3946bd956afacf695fabaa P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:43.951315 1064 raft_consensus.cc:2454] T 2ecebe8dad3946bd956afacf695fabaa P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27ec1b26c2f64cc682ed7ae12e133a9f in term 2.
I20240502 07:59:43.960629 956 catalog_manager.cc:5484] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f reported cstate change: term changed from 1 to 2, leader changed from 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129) to 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131). New cstate: current_term: 2 leader_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: HEALTHY } } }
I20240502 07:59:43.960929 956 catalog_manager.cc:5484] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f reported cstate change: term changed from 1 to 2, leader changed from b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130) to 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131). New cstate: current_term: 2 leader_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: HEALTHY } } }
W20240502 07:59:43.964725 1119 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47900: Illegal state: replica b1e9945160714b16b2db91f90c5ccc70 is not leader of this config: current role FOLLOWER
I20240502 07:59:43.965150 1064 raft_consensus.cc:1226] T 682e26b333104f82863b92c545cfdafe P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Rejecting Update request from peer b1e9945160714b16b2db91f90c5ccc70 for earlier term 1. Current term is 2. Ops: [1.22-1.22]
I20240502 07:59:43.965517 1064 raft_consensus.cc:1226] T 2ecebe8dad3946bd956afacf695fabaa P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Rejecting Update request from peer b1e9945160714b16b2db91f90c5ccc70 for earlier term 1. Current term is 2. Ops: [1.22-1.22]
W20240502 07:59:43.967206 1119 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47900: Illegal state: replica b1e9945160714b16b2db91f90c5ccc70 is not leader of this config: current role FOLLOWER
W20240502 07:59:43.967705 1119 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47900: Illegal state: replica b1e9945160714b16b2db91f90c5ccc70 is not leader of this config: current role FOLLOWER
W20240502 07:59:43.968192 1119 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47900: Illegal state: replica b1e9945160714b16b2db91f90c5ccc70 is not leader of this config: current role FOLLOWER
W20240502 07:59:43.974766 1042 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48328: Illegal state: replica 77e648af9dbb46fc881340efe1d1e3c3 is not leader of this config: current role FOLLOWER
W20240502 07:59:43.978132 1042 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48328: Illegal state: replica 77e648af9dbb46fc881340efe1d1e3c3 is not leader of this config: current role FOLLOWER
W20240502 07:59:43.978704 1042 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48328: Illegal state: replica 77e648af9dbb46fc881340efe1d1e3c3 is not leader of this config: current role FOLLOWER
W20240502 07:59:43.979394 1040 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48328: Illegal state: replica 77e648af9dbb46fc881340efe1d1e3c3 is not leader of this config: current role FOLLOWER
W20240502 07:59:43.979414 1039 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48328: Illegal state: replica 77e648af9dbb46fc881340efe1d1e3c3 is not leader of this config: current role FOLLOWER
W20240502 07:59:43.979794 1045 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48328: Illegal state: replica 77e648af9dbb46fc881340efe1d1e3c3 is not leader of this config: current role FOLLOWER
W20240502 07:59:43.979794 1038 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48328: Illegal state: replica 77e648af9dbb46fc881340efe1d1e3c3 is not leader of this config: current role FOLLOWER
W20240502 07:59:43.980281 1039 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48328: Illegal state: replica 77e648af9dbb46fc881340efe1d1e3c3 is not leader of this config: current role FOLLOWER
W20240502 07:59:43.980798 1040 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48328: Illegal state: replica 77e648af9dbb46fc881340efe1d1e3c3 is not leader of this config: current role FOLLOWER
W20240502 07:59:43.981279 1042 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48328: Illegal state: replica 77e648af9dbb46fc881340efe1d1e3c3 is not leader of this config: current role FOLLOWER
I20240502 07:59:43.984764 1139 raft_consensus.cc:1261] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 2 index: 24. (index mismatch)
I20240502 07:59:43.985141 1063 raft_consensus.cc:1261] T 682e26b333104f82863b92c545cfdafe P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 1 index: 21. Preceding OpId from leader: term: 2 index: 24. (index mismatch)
I20240502 07:59:43.985440 1325 consensus_queue.cc:1035] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 19, Time since last communication: 0.000s
I20240502 07:59:43.985558 1276 consensus_queue.cc:1035] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20240502 07:59:43.987008 1063 raft_consensus.cc:1261] T 2ecebe8dad3946bd956afacf695fabaa P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 1 index: 21. Preceding OpId from leader: term: 2 index: 24. (index mismatch)
I20240502 07:59:43.987365 1400 consensus_queue.cc:1035] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 18, Time since last communication: 0.000s
I20240502 07:59:43.993849 1143 raft_consensus.cc:1261] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 2 index: 24. (index mismatch)
I20240502 07:59:44.001019 1276 consensus_queue.cc:1035] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
W20240502 07:59:44.019590 1119 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47900: Illegal state: replica b1e9945160714b16b2db91f90c5ccc70 is not leader of this config: current role FOLLOWER
W20240502 07:59:44.020364 1123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47900: Illegal state: replica b1e9945160714b16b2db91f90c5ccc70 is not leader of this config: current role FOLLOWER
W20240502 07:59:44.021047 1118 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47900: Illegal state: replica b1e9945160714b16b2db91f90c5ccc70 is not leader of this config: current role FOLLOWER
W20240502 07:59:44.021579 1123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47900: Illegal state: replica b1e9945160714b16b2db91f90c5ccc70 is not leader of this config: current role FOLLOWER
I20240502 07:59:44.026401 1066 raft_consensus.cc:1261] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 07:59:44.027607 1325 consensus_queue.cc:1035] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240502 07:59:44.028014 1141 raft_consensus.cc:1261] T 69a9ebd0c390436b8ace08089f719358 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 07:59:44.030397 1400 consensus_queue.cc:1035] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.000s
I20240502 07:59:44.041045 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:44.043967 1408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:44.044799 1409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:44.045627 1411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:44.046250 30122 server_base.cc:1034] running on GCE node
I20240502 07:59:44.046435 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:59:44.046478 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:59:44.046494 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636784046494 us; error 0 us; skew 500 ppm
I20240502 07:59:44.047605 30122 webserver.cc:460] Webserver started at http://127.29.106.132:34149/ using document root <none> and password file <none>
I20240502 07:59:44.047766 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:44.047823 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:44.047889 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:44.048346 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-3-root/instance:
uuid: "9d994926b3d54e6395a0d60df8a1db07"
format_stamp: "Formatted at 2024-05-02 07:59:44 on dist-test-slave-m6l2"
I20240502 07:59:44.050751 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:44.051751 1416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:44.052101 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:44.052171 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-3-root
uuid: "9d994926b3d54e6395a0d60df8a1db07"
format_stamp: "Formatted at 2024-05-02 07:59:44 on dist-test-slave-m6l2"
I20240502 07:59:44.052256 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:44.087745 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:44.088181 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:44.088496 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:44.088945 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:44.088986 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:44.089018 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:44.089033 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:44.093957 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.132:44697
I20240502 07:59:44.093991 1479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.132:44697 every 8 connection(s)
I20240502 07:59:44.094812 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:44.097705 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:44.098318 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:44.099292 1480 heartbeater.cc:347] Connected to a master server at 127.29.106.190:35263
I20240502 07:59:44.099406 1480 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:44.099568 30122 server_base.cc:1034] running on GCE node
W20240502 07:59:44.099627 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:44.099689 1480 heartbeater.cc:511] Master 127.29.106.190:35263 requested a full tablet report, sending...
I20240502 07:59:44.099882 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:59:44.099933 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:59:44.099953 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636784099953 us; error 0 us; skew 500 ppm
I20240502 07:59:44.100379 962 ts_manager.cc:194] Registered new tserver with Master: 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132:44697)
I20240502 07:59:44.100917 962 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40032
I20240502 07:59:44.101110 30122 webserver.cc:460] Webserver started at http://127.29.106.133:38455/ using document root <none> and password file <none>
I20240502 07:59:44.101279 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:44.101332 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:44.101382 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:44.101750 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-4-root/instance:
uuid: "66df61031360493886f1170a809e079a"
format_stamp: "Formatted at 2024-05-02 07:59:44 on dist-test-slave-m6l2"
I20240502 07:59:44.103165 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 07:59:44.104139 1492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:44.104413 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:44.104481 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-4-root
uuid: "66df61031360493886f1170a809e079a"
format_stamp: "Formatted at 2024-05-02 07:59:44 on dist-test-slave-m6l2"
I20240502 07:59:44.104550 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-4-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-4-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-4-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:44.117257 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:44.117698 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:44.117977 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:44.118407 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:44.118451 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:44.118484 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:44.118515 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:44.123152 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.133:38717
I20240502 07:59:44.123266 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.133:38717 every 8 connection(s)
I20240502 07:59:44.124084 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:44.126736 1560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:44.127414 1562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:44.128015 1559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:44.128172 1556 heartbeater.cc:347] Connected to a master server at 127.29.106.190:35263
I20240502 07:59:44.128300 1556 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:44.128536 1556 heartbeater.cc:511] Master 127.29.106.190:35263 requested a full tablet report, sending...
I20240502 07:59:44.128692 30122 server_base.cc:1034] running on GCE node
I20240502 07:59:44.128860 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 07:59:44.128911 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 07:59:44.128949 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636784128948 us; error 0 us; skew 500 ppm
I20240502 07:59:44.129187 962 ts_manager.cc:194] Registered new tserver with Master: 66df61031360493886f1170a809e079a (127.29.106.133:38717)
I20240502 07:59:44.129827 962 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40036
I20240502 07:59:44.130112 30122 webserver.cc:460] Webserver started at http://127.29.106.134:43249/ using document root <none> and password file <none>
I20240502 07:59:44.130241 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:44.130301 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:44.130374 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:44.130843 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-5-root/instance:
uuid: "eb0a49d2a951470e85adbcd797128e69"
format_stamp: "Formatted at 2024-05-02 07:59:44 on dist-test-slave-m6l2"
I20240502 07:59:44.132252 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 07:59:44.133236 1567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:44.133522 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:44.133586 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-5-root
uuid: "eb0a49d2a951470e85adbcd797128e69"
format_stamp: "Formatted at 2024-05-02 07:59:44 on dist-test-slave-m6l2"
I20240502 07:59:44.133668 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-5-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-5-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-5-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:44.149006 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:44.149492 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:44.149847 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:44.150355 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:44.150400 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:44.150442 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:44.150478 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:44.155939 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.134:34541
I20240502 07:59:44.156107 1630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.134:34541 every 8 connection(s)
I20240502 07:59:44.160650 1631 heartbeater.cc:347] Connected to a master server at 127.29.106.190:35263
I20240502 07:59:44.160755 1631 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:44.160975 1631 heartbeater.cc:511] Master 127.29.106.190:35263 requested a full tablet report, sending...
I20240502 07:59:44.161553 962 ts_manager.cc:194] Registered new tserver with Master: eb0a49d2a951470e85adbcd797128e69 (127.29.106.134:34541)
I20240502 07:59:44.162088 962 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40040
I20240502 07:59:44.814800 1141 consensus_queue.cc:237] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } }
I20240502 07:59:44.816366 1063 raft_consensus.cc:1261] T 6edf17550e3f4a0b945477edc6de6af1 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 07:59:44.816758 1218 raft_consensus.cc:1261] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 07:59:44.816849 1637 consensus_queue.cc:1035] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 07:59:44.817191 1638 consensus_queue.cc:1035] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 07:59:44.818655 1637 raft_consensus.cc:2932] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } } }
I20240502 07:59:44.819343 1218 raft_consensus.cc:2932] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } } }
W20240502 07:59:44.821141 1109 consensus_peers.cc:485] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 -> Peer eb0a49d2a951470e85adbcd797128e69 (127.29.106.134:34541): Couldn't send request to peer eb0a49d2a951470e85adbcd797128e69. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6edf17550e3f4a0b945477edc6de6af1. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:44.821246 1063 raft_consensus.cc:2932] T 6edf17550e3f4a0b945477edc6de6af1 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } } }
I20240502 07:59:44.822149 962 catalog_manager.cc:5484] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f reported cstate change: config changed from index -1 to 29, NON_VOTER eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) added. New cstate: current_term: 1 leader_uuid: "b1e9945160714b16b2db91f90c5ccc70" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } } }
I20240502 07:59:44.827088 1063 consensus_queue.cc:237] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } }
I20240502 07:59:44.828316 1141 raft_consensus.cc:1261] T fda98cfe14f947969fb41e26c75bc384 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 07:59:44.828563 1218 raft_consensus.cc:1261] T fda98cfe14f947969fb41e26c75bc384 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 07:59:44.828943 1646 consensus_queue.cc:1035] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 07:59:44.829171 1645 consensus_queue.cc:1035] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 07:59:44.830799 1376 raft_consensus.cc:2932] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } } }
I20240502 07:59:44.831276 1141 raft_consensus.cc:2932] T fda98cfe14f947969fb41e26c75bc384 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } } }
I20240502 07:59:44.833465 1218 raft_consensus.cc:2932] T fda98cfe14f947969fb41e26c75bc384 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } } }
I20240502 07:59:44.834013 960 catalog_manager.cc:5484] T fda98cfe14f947969fb41e26c75bc384 P b1e9945160714b16b2db91f90c5ccc70 reported cstate change: config changed from index -1 to 29, NON_VOTER 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) added. New cstate: current_term: 1 leader_uuid: "77e648af9dbb46fc881340efe1d1e3c3" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } } }
W20240502 07:59:44.835134 1030 consensus_peers.cc:485] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 -> Peer 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132:44697): Couldn't send request to peer 9d994926b3d54e6395a0d60df8a1db07. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fda98cfe14f947969fb41e26c75bc384. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:44.836556 1063 consensus_queue.cc:237] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } }
I20240502 07:59:44.838105 1218 raft_consensus.cc:1261] T c7c1536618ae4e74aa79349d04201fb9 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 07:59:44.838339 1141 raft_consensus.cc:1261] T c7c1536618ae4e74aa79349d04201fb9 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 07:59:44.838478 1645 consensus_queue.cc:1035] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 07:59:44.838838 1645 consensus_queue.cc:1035] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 07:59:44.839952 1646 raft_consensus.cc:2932] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 66df61031360493886f1170a809e079a (127.29.106.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } } }
I20240502 07:59:44.840328 1218 raft_consensus.cc:2932] T c7c1536618ae4e74aa79349d04201fb9 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 66df61031360493886f1170a809e079a (127.29.106.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } } }
I20240502 07:59:44.842501 962 catalog_manager.cc:5484] T c7c1536618ae4e74aa79349d04201fb9 P 27ec1b26c2f64cc682ed7ae12e133a9f reported cstate change: config changed from index -1 to 29, NON_VOTER 66df61031360493886f1170a809e079a (127.29.106.133) added. New cstate: current_term: 1 leader_uuid: "77e648af9dbb46fc881340efe1d1e3c3" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } } }
W20240502 07:59:44.842967 1033 consensus_peers.cc:485] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 -> Peer 66df61031360493886f1170a809e079a (127.29.106.133:38717): Couldn't send request to peer 66df61031360493886f1170a809e079a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c7c1536618ae4e74aa79349d04201fb9. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:44.843204 1141 raft_consensus.cc:2932] T c7c1536618ae4e74aa79349d04201fb9 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 66df61031360493886f1170a809e079a (127.29.106.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } } }
I20240502 07:59:44.846243 1063 consensus_queue.cc:237] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } }
I20240502 07:59:44.847832 1218 raft_consensus.cc:1261] T df4233b549b34b01b6f9662c7e7b3515 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 07:59:44.847827 1141 raft_consensus.cc:1261] T df4233b549b34b01b6f9662c7e7b3515 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
W20240502 07:59:44.848141 1033 consensus_peers.cc:485] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 -> Peer 66df61031360493886f1170a809e079a (127.29.106.133:38717): Couldn't send request to peer 66df61031360493886f1170a809e079a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df4233b549b34b01b6f9662c7e7b3515. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:44.848366 1650 consensus_queue.cc:1035] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 07:59:44.848583 1646 consensus_queue.cc:1035] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 07:59:44.850317 1376 raft_consensus.cc:2932] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 66df61031360493886f1170a809e079a (127.29.106.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } } }
I20240502 07:59:44.850798 1141 raft_consensus.cc:2932] T df4233b549b34b01b6f9662c7e7b3515 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 66df61031360493886f1170a809e079a (127.29.106.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } } }
I20240502 07:59:44.852377 1218 raft_consensus.cc:2932] T df4233b549b34b01b6f9662c7e7b3515 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 66df61031360493886f1170a809e079a (127.29.106.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } } }
I20240502 07:59:44.852684 956 catalog_manager.cc:5484] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 reported cstate change: config changed from index -1 to 29, NON_VOTER 66df61031360493886f1170a809e079a (127.29.106.133) added. New cstate: current_term: 1 leader_uuid: "77e648af9dbb46fc881340efe1d1e3c3" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:44.859987 1218 consensus_queue.cc:237] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 24, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } }
I20240502 07:59:44.861685 1063 raft_consensus.cc:1261] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 07:59:44.861743 1141 raft_consensus.cc:1261] T 69a9ebd0c390436b8ace08089f719358 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 07:59:44.862165 1652 consensus_queue.cc:1035] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:44.862495 1407 consensus_queue.cc:1035] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:44.864105 1407 raft_consensus.cc:2932] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index -1 to 30, NON_VOTER eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } } }
I20240502 07:59:44.864727 1063 raft_consensus.cc:2932] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index -1 to 30, NON_VOTER eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } } }
I20240502 07:59:44.864732 1141 raft_consensus.cc:2932] T 69a9ebd0c390436b8ace08089f719358 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index -1 to 30, NON_VOTER eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } } }
I20240502 07:59:44.867575 1143 consensus_queue.cc:237] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } }
I20240502 07:59:44.867616 956 catalog_manager.cc:5484] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 reported cstate change: config changed from index -1 to 30, NON_VOTER eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) added. New cstate: current_term: 2 leader_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } } }
W20240502 07:59:44.869123 1184 consensus_peers.cc:485] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f -> Peer eb0a49d2a951470e85adbcd797128e69 (127.29.106.134:34541): Couldn't send request to peer eb0a49d2a951470e85adbcd797128e69. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69a9ebd0c390436b8ace08089f719358. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:44.870512 1063 raft_consensus.cc:1261] T 2c1b6523d6724a02acc9ae19f3f0255e P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 07:59:44.870622 1218 raft_consensus.cc:1261] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 07:59:44.871407 1314 consensus_queue.cc:1035] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 07:59:44.873149 1314 consensus_queue.cc:1035] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 07:59:44.875751 1638 raft_consensus.cc:2932] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } } }
I20240502 07:59:44.876587 1063 raft_consensus.cc:2932] T 2c1b6523d6724a02acc9ae19f3f0255e P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } } }
I20240502 07:59:44.877193 1218 raft_consensus.cc:2932] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } } }
I20240502 07:59:44.879034 956 catalog_manager.cc:5484] T 2c1b6523d6724a02acc9ae19f3f0255e P 77e648af9dbb46fc881340efe1d1e3c3 reported cstate change: config changed from index -1 to 29, NON_VOTER 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) added. New cstate: current_term: 1 leader_uuid: "b1e9945160714b16b2db91f90c5ccc70" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } } }
W20240502 07:59:44.880535 1106 consensus_peers.cc:485] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 -> Peer 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132:44697): Couldn't send request to peer 9d994926b3d54e6395a0d60df8a1db07. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c1b6523d6724a02acc9ae19f3f0255e. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:45.101769 1480 heartbeater.cc:503] Master 127.29.106.190:35263 was elected leader, sending a full tablet report...
I20240502 07:59:45.130879 1556 heartbeater.cc:503] Master 127.29.106.190:35263 was elected leader, sending a full tablet report...
I20240502 07:59:45.162899 1631 heartbeater.cc:503] Master 127.29.106.190:35263 was elected leader, sending a full tablet report...
I20240502 07:59:45.247340 1654 ts_tablet_manager.cc:902] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07: Initiating tablet copy from peer b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203)
I20240502 07:59:45.248025 1654 tablet_copy_client.cc:308] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07: tablet copy: Beginning tablet copy session from remote peer at address 127.29.106.130:38203
I20240502 07:59:45.251952 1153 tablet_copy_service.cc:140] P b1e9945160714b16b2db91f90c5ccc70: Received BeginTabletCopySession request for tablet 2c1b6523d6724a02acc9ae19f3f0255e from peer 9d994926b3d54e6395a0d60df8a1db07 ({username='slave'} at 127.0.0.1:47922)
I20240502 07:59:45.252071 1153 tablet_copy_service.cc:161] P b1e9945160714b16b2db91f90c5ccc70: Beginning new tablet copy session on tablet 2c1b6523d6724a02acc9ae19f3f0255e from peer 9d994926b3d54e6395a0d60df8a1db07 at {username='slave'} at 127.0.0.1:47922: session id = 9d994926b3d54e6395a0d60df8a1db07-2c1b6523d6724a02acc9ae19f3f0255e
I20240502 07:59:45.253484 1153 tablet_copy_source_session.cc:205] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:45.254740 1654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c1b6523d6724a02acc9ae19f3f0255e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:45.257337 1654 tablet_copy_client.cc:783] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:45.257632 1654 tablet_copy_client.cc:647] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:45.258968 1657 tablet_copy_client.cc:1002] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07: tablet copy: Injecting latency into file download: 2000
I20240502 07:59:45.286415 1660 ts_tablet_manager.cc:902] T fda98cfe14f947969fb41e26c75bc384 P 9d994926b3d54e6395a0d60df8a1db07: Initiating tablet copy from peer 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761)
I20240502 07:59:45.287055 1660 tablet_copy_client.cc:308] T fda98cfe14f947969fb41e26c75bc384 P 9d994926b3d54e6395a0d60df8a1db07: tablet copy: Beginning tablet copy session from remote peer at address 127.29.106.129:34761
I20240502 07:59:45.291523 1077 tablet_copy_service.cc:140] P 77e648af9dbb46fc881340efe1d1e3c3: Received BeginTabletCopySession request for tablet fda98cfe14f947969fb41e26c75bc384 from peer 9d994926b3d54e6395a0d60df8a1db07 ({username='slave'} at 127.0.0.1:48350)
I20240502 07:59:45.291668 1077 tablet_copy_service.cc:161] P 77e648af9dbb46fc881340efe1d1e3c3: Beginning new tablet copy session on tablet fda98cfe14f947969fb41e26c75bc384 from peer 9d994926b3d54e6395a0d60df8a1db07 at {username='slave'} at 127.0.0.1:48350: session id = 9d994926b3d54e6395a0d60df8a1db07-fda98cfe14f947969fb41e26c75bc384
I20240502 07:59:45.292851 1077 tablet_copy_source_session.cc:205] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:45.293753 1660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fda98cfe14f947969fb41e26c75bc384. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:45.296190 1660 tablet_copy_client.cc:783] T fda98cfe14f947969fb41e26c75bc384 P 9d994926b3d54e6395a0d60df8a1db07: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:45.296401 1660 tablet_copy_client.cc:647] T fda98cfe14f947969fb41e26c75bc384 P 9d994926b3d54e6395a0d60df8a1db07: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:45.297752 1664 tablet_copy_client.cc:1002] T fda98cfe14f947969fb41e26c75bc384 P 9d994926b3d54e6395a0d60df8a1db07: tablet copy: Injecting latency into file download: 2000
I20240502 07:59:45.299968 1665 ts_tablet_manager.cc:902] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69: Initiating tablet copy from peer 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:45.301158 1665 tablet_copy_client.cc:308] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69: tablet copy: Beginning tablet copy session from remote peer at address 127.29.106.131:36737
I20240502 07:59:45.306012 1228 tablet_copy_service.cc:140] P 27ec1b26c2f64cc682ed7ae12e133a9f: Received BeginTabletCopySession request for tablet 69a9ebd0c390436b8ace08089f719358 from peer eb0a49d2a951470e85adbcd797128e69 ({username='slave'} at 127.0.0.1:52082)
I20240502 07:59:45.306286 1228 tablet_copy_service.cc:161] P 27ec1b26c2f64cc682ed7ae12e133a9f: Beginning new tablet copy session on tablet 69a9ebd0c390436b8ace08089f719358 from peer eb0a49d2a951470e85adbcd797128e69 at {username='slave'} at 127.0.0.1:52082: session id = eb0a49d2a951470e85adbcd797128e69-69a9ebd0c390436b8ace08089f719358
I20240502 07:59:45.307969 1228 tablet_copy_source_session.cc:205] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:45.308952 1665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69a9ebd0c390436b8ace08089f719358. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:45.311555 1665 tablet_copy_client.cc:783] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:45.311745 1665 tablet_copy_client.cc:647] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:45.313014 1669 tablet_copy_client.cc:1002] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69: tablet copy: Injecting latency into file download: 2000
I20240502 07:59:45.337548 1671 ts_tablet_manager.cc:902] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69: Initiating tablet copy from peer b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203)
I20240502 07:59:45.338174 1671 tablet_copy_client.cc:308] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69: tablet copy: Beginning tablet copy session from remote peer at address 127.29.106.130:38203
I20240502 07:59:45.341852 1676 ts_tablet_manager.cc:902] T c7c1536618ae4e74aa79349d04201fb9 P 66df61031360493886f1170a809e079a: Initiating tablet copy from peer 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761)
I20240502 07:59:45.342044 1153 tablet_copy_service.cc:140] P b1e9945160714b16b2db91f90c5ccc70: Received BeginTabletCopySession request for tablet 6edf17550e3f4a0b945477edc6de6af1 from peer eb0a49d2a951470e85adbcd797128e69 ({username='slave'} at 127.0.0.1:47924)
I20240502 07:59:45.342242 1153 tablet_copy_service.cc:161] P b1e9945160714b16b2db91f90c5ccc70: Beginning new tablet copy session on tablet 6edf17550e3f4a0b945477edc6de6af1 from peer eb0a49d2a951470e85adbcd797128e69 at {username='slave'} at 127.0.0.1:47924: session id = eb0a49d2a951470e85adbcd797128e69-6edf17550e3f4a0b945477edc6de6af1
I20240502 07:59:45.342481 1676 tablet_copy_client.cc:308] T c7c1536618ae4e74aa79349d04201fb9 P 66df61031360493886f1170a809e079a: tablet copy: Beginning tablet copy session from remote peer at address 127.29.106.129:34761
I20240502 07:59:45.343449 1153 tablet_copy_source_session.cc:205] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:45.344504 1671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6edf17550e3f4a0b945477edc6de6af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:45.346872 1671 tablet_copy_client.cc:783] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:45.347010 1077 tablet_copy_service.cc:140] P 77e648af9dbb46fc881340efe1d1e3c3: Received BeginTabletCopySession request for tablet c7c1536618ae4e74aa79349d04201fb9 from peer 66df61031360493886f1170a809e079a ({username='slave'} at 127.0.0.1:48364)
I20240502 07:59:45.347121 1077 tablet_copy_service.cc:161] P 77e648af9dbb46fc881340efe1d1e3c3: Beginning new tablet copy session on tablet c7c1536618ae4e74aa79349d04201fb9 from peer 66df61031360493886f1170a809e079a at {username='slave'} at 127.0.0.1:48364: session id = 66df61031360493886f1170a809e079a-c7c1536618ae4e74aa79349d04201fb9
I20240502 07:59:45.347046 1671 tablet_copy_client.cc:647] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:45.348146 1077 tablet_copy_source_session.cc:205] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:45.348471 1673 tablet_copy_client.cc:1002] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69: tablet copy: Injecting latency into file download: 2000
I20240502 07:59:45.348887 1676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7c1536618ae4e74aa79349d04201fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:45.350754 1676 tablet_copy_client.cc:783] T c7c1536618ae4e74aa79349d04201fb9 P 66df61031360493886f1170a809e079a: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:45.350912 1676 tablet_copy_client.cc:647] T c7c1536618ae4e74aa79349d04201fb9 P 66df61031360493886f1170a809e079a: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:45.351889 1678 tablet_copy_client.cc:1002] T c7c1536618ae4e74aa79349d04201fb9 P 66df61031360493886f1170a809e079a: tablet copy: Injecting latency into file download: 2000
I20240502 07:59:45.457268 1682 ts_tablet_manager.cc:902] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a: Initiating tablet copy from peer 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761)
I20240502 07:59:45.457988 1682 tablet_copy_client.cc:308] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a: tablet copy: Beginning tablet copy session from remote peer at address 127.29.106.129:34761
I20240502 07:59:45.458482 1077 tablet_copy_service.cc:140] P 77e648af9dbb46fc881340efe1d1e3c3: Received BeginTabletCopySession request for tablet df4233b549b34b01b6f9662c7e7b3515 from peer 66df61031360493886f1170a809e079a ({username='slave'} at 127.0.0.1:48364)
I20240502 07:59:45.458595 1077 tablet_copy_service.cc:161] P 77e648af9dbb46fc881340efe1d1e3c3: Beginning new tablet copy session on tablet df4233b549b34b01b6f9662c7e7b3515 from peer 66df61031360493886f1170a809e079a at {username='slave'} at 127.0.0.1:48364: session id = 66df61031360493886f1170a809e079a-df4233b549b34b01b6f9662c7e7b3515
I20240502 07:59:45.460031 1077 tablet_copy_source_session.cc:205] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:45.461371 1682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df4233b549b34b01b6f9662c7e7b3515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:45.463443 1682 tablet_copy_client.cc:783] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:45.463704 1682 tablet_copy_client.cc:647] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:45.464730 1684 tablet_copy_client.cc:1002] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a: tablet copy: Injecting latency into file download: 2000
I20240502 07:59:45.916603 1018 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:59:45.918048 1018 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 07:59:45.918171 1018 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:59:47.259307 1654 tablet_copy_client.cc:523] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:47.260870 1654 tablet_bootstrap.cc:492] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07: Bootstrap starting.
I20240502 07:59:47.289856 1654 tablet_bootstrap.cc:492] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=136 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:47.290190 1654 tablet_bootstrap.cc:492] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07: Bootstrap complete.
I20240502 07:59:47.290410 1654 ts_tablet_manager.cc:1367] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07: Time spent bootstrapping tablet: real 0.030s user 0.025s sys 0.003s
I20240502 07:59:47.290944 1654 raft_consensus.cc:348] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } }
I20240502 07:59:47.291074 1654 raft_consensus.cc:729] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9d994926b3d54e6395a0d60df8a1db07, State: Initialized, Role: LEARNER
I20240502 07:59:47.291169 1654 consensus_queue.cc:260] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } }
I20240502 07:59:47.291589 1654 ts_tablet_manager.cc:1398] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 07:59:47.292215 1153 tablet_copy_service.cc:342] P b1e9945160714b16b2db91f90c5ccc70: Request end of tablet copy session 9d994926b3d54e6395a0d60df8a1db07-2c1b6523d6724a02acc9ae19f3f0255e received from {username='slave'} at 127.0.0.1:47922
I20240502 07:59:47.292405 1153 tablet_copy_service.cc:433] P b1e9945160714b16b2db91f90c5ccc70: ending tablet copy session 9d994926b3d54e6395a0d60df8a1db07-2c1b6523d6724a02acc9ae19f3f0255e on tablet 2c1b6523d6724a02acc9ae19f3f0255e with peer 9d994926b3d54e6395a0d60df8a1db07
I20240502 07:59:47.298456 1660 tablet_copy_client.cc:523] T fda98cfe14f947969fb41e26c75bc384 P 9d994926b3d54e6395a0d60df8a1db07: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:47.300118 1660 tablet_bootstrap.cc:492] T fda98cfe14f947969fb41e26c75bc384 P 9d994926b3d54e6395a0d60df8a1db07: Bootstrap starting.
I20240502 07:59:47.313344 1665 tablet_copy_client.cc:523] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:47.314795 1665 tablet_bootstrap.cc:492] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69: Bootstrap starting.
I20240502 07:59:47.324169 1660 tablet_bootstrap.cc:492] T fda98cfe14f947969fb41e26c75bc384 P 9d994926b3d54e6395a0d60df8a1db07: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=110 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:47.324595 1660 tablet_bootstrap.cc:492] T fda98cfe14f947969fb41e26c75bc384 P 9d994926b3d54e6395a0d60df8a1db07: Bootstrap complete.
I20240502 07:59:47.324824 1660 ts_tablet_manager.cc:1367] T fda98cfe14f947969fb41e26c75bc384 P 9d994926b3d54e6395a0d60df8a1db07: Time spent bootstrapping tablet: real 0.025s user 0.022s sys 0.004s
I20240502 07:59:47.325408 1660 raft_consensus.cc:348] T fda98cfe14f947969fb41e26c75bc384 P 9d994926b3d54e6395a0d60df8a1db07 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } }
I20240502 07:59:47.325565 1660 raft_consensus.cc:729] T fda98cfe14f947969fb41e26c75bc384 P 9d994926b3d54e6395a0d60df8a1db07 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9d994926b3d54e6395a0d60df8a1db07, State: Initialized, Role: LEARNER
I20240502 07:59:47.325699 1660 consensus_queue.cc:260] T fda98cfe14f947969fb41e26c75bc384 P 9d994926b3d54e6395a0d60df8a1db07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } }
I20240502 07:59:47.326014 1660 ts_tablet_manager.cc:1398] T fda98cfe14f947969fb41e26c75bc384 P 9d994926b3d54e6395a0d60df8a1db07: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:47.326602 1077 tablet_copy_service.cc:342] P 77e648af9dbb46fc881340efe1d1e3c3: Request end of tablet copy session 9d994926b3d54e6395a0d60df8a1db07-fda98cfe14f947969fb41e26c75bc384 received from {username='slave'} at 127.0.0.1:48350
I20240502 07:59:47.326689 1077 tablet_copy_service.cc:433] P 77e648af9dbb46fc881340efe1d1e3c3: ending tablet copy session 9d994926b3d54e6395a0d60df8a1db07-fda98cfe14f947969fb41e26c75bc384 on tablet fda98cfe14f947969fb41e26c75bc384 with peer 9d994926b3d54e6395a0d60df8a1db07
I20240502 07:59:47.341066 1665 tablet_bootstrap.cc:492] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=118 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:47.341358 1665 tablet_bootstrap.cc:492] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69: Bootstrap complete.
I20240502 07:59:47.341559 1665 ts_tablet_manager.cc:1367] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69: Time spent bootstrapping tablet: real 0.027s user 0.022s sys 0.004s
I20240502 07:59:47.342092 1665 raft_consensus.cc:348] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } }
I20240502 07:59:47.342207 1665 raft_consensus.cc:729] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: eb0a49d2a951470e85adbcd797128e69, State: Initialized, Role: LEARNER
I20240502 07:59:47.342319 1665 consensus_queue.cc:260] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } }
I20240502 07:59:47.342690 1665 ts_tablet_manager.cc:1398] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:59:47.343400 1228 tablet_copy_service.cc:342] P 27ec1b26c2f64cc682ed7ae12e133a9f: Request end of tablet copy session eb0a49d2a951470e85adbcd797128e69-69a9ebd0c390436b8ace08089f719358 received from {username='slave'} at 127.0.0.1:52082
I20240502 07:59:47.343539 1228 tablet_copy_service.cc:433] P 27ec1b26c2f64cc682ed7ae12e133a9f: ending tablet copy session eb0a49d2a951470e85adbcd797128e69-69a9ebd0c390436b8ace08089f719358 on tablet 69a9ebd0c390436b8ace08089f719358 with peer eb0a49d2a951470e85adbcd797128e69
I20240502 07:59:47.348825 1671 tablet_copy_client.cc:523] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:47.350464 1671 tablet_bootstrap.cc:492] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69: Bootstrap starting.
I20240502 07:59:47.352196 1676 tablet_copy_client.cc:523] T c7c1536618ae4e74aa79349d04201fb9 P 66df61031360493886f1170a809e079a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:47.354269 1676 tablet_bootstrap.cc:492] T c7c1536618ae4e74aa79349d04201fb9 P 66df61031360493886f1170a809e079a: Bootstrap starting.
I20240502 07:59:47.374763 1671 tablet_bootstrap.cc:492] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=110 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:47.375119 1671 tablet_bootstrap.cc:492] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69: Bootstrap complete.
I20240502 07:59:47.375337 1671 ts_tablet_manager.cc:1367] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69: Time spent bootstrapping tablet: real 0.025s user 0.023s sys 0.000s
I20240502 07:59:47.375891 1671 raft_consensus.cc:348] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } }
I20240502 07:59:47.376036 1671 raft_consensus.cc:729] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb0a49d2a951470e85adbcd797128e69, State: Initialized, Role: LEARNER
I20240502 07:59:47.376147 1671 consensus_queue.cc:260] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } }
I20240502 07:59:47.376514 1671 ts_tablet_manager.cc:1398] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:47.377276 1153 tablet_copy_service.cc:342] P b1e9945160714b16b2db91f90c5ccc70: Request end of tablet copy session eb0a49d2a951470e85adbcd797128e69-6edf17550e3f4a0b945477edc6de6af1 received from {username='slave'} at 127.0.0.1:47924
I20240502 07:59:47.377367 1153 tablet_copy_service.cc:433] P b1e9945160714b16b2db91f90c5ccc70: ending tablet copy session eb0a49d2a951470e85adbcd797128e69-6edf17550e3f4a0b945477edc6de6af1 on tablet 6edf17550e3f4a0b945477edc6de6af1 with peer eb0a49d2a951470e85adbcd797128e69
I20240502 07:59:47.381421 1605 raft_consensus.cc:1203] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29] Dedup: 1.29->[]
I20240502 07:59:47.386453 1676 tablet_bootstrap.cc:492] T c7c1536618ae4e74aa79349d04201fb9 P 66df61031360493886f1170a809e079a: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=122 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:47.386812 1676 tablet_bootstrap.cc:492] T c7c1536618ae4e74aa79349d04201fb9 P 66df61031360493886f1170a809e079a: Bootstrap complete.
I20240502 07:59:47.387082 1676 ts_tablet_manager.cc:1367] T c7c1536618ae4e74aa79349d04201fb9 P 66df61031360493886f1170a809e079a: Time spent bootstrapping tablet: real 0.033s user 0.028s sys 0.004s
I20240502 07:59:47.387887 1676 raft_consensus.cc:348] T c7c1536618ae4e74aa79349d04201fb9 P 66df61031360493886f1170a809e079a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } }
I20240502 07:59:47.388013 1676 raft_consensus.cc:729] T c7c1536618ae4e74aa79349d04201fb9 P 66df61031360493886f1170a809e079a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66df61031360493886f1170a809e079a, State: Initialized, Role: LEARNER
I20240502 07:59:47.388104 1676 consensus_queue.cc:260] T c7c1536618ae4e74aa79349d04201fb9 P 66df61031360493886f1170a809e079a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } }
I20240502 07:59:47.388510 1676 ts_tablet_manager.cc:1398] T c7c1536618ae4e74aa79349d04201fb9 P 66df61031360493886f1170a809e079a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:47.389328 1077 tablet_copy_service.cc:342] P 77e648af9dbb46fc881340efe1d1e3c3: Request end of tablet copy session 66df61031360493886f1170a809e079a-c7c1536618ae4e74aa79349d04201fb9 received from {username='slave'} at 127.0.0.1:48364
I20240502 07:59:47.389402 1077 tablet_copy_service.cc:433] P 77e648af9dbb46fc881340efe1d1e3c3: ending tablet copy session 66df61031360493886f1170a809e079a-c7c1536618ae4e74aa79349d04201fb9 on tablet c7c1536618ae4e74aa79349d04201fb9 with peer 66df61031360493886f1170a809e079a
I20240502 07:59:47.464993 1682 tablet_copy_client.cc:523] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:47.466574 1682 tablet_bootstrap.cc:492] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a: Bootstrap starting.
I20240502 07:59:47.489146 1682 tablet_bootstrap.cc:492] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=126 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:47.489449 1682 tablet_bootstrap.cc:492] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a: Bootstrap complete.
I20240502 07:59:47.489646 1682 ts_tablet_manager.cc:1367] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a: Time spent bootstrapping tablet: real 0.023s user 0.023s sys 0.000s
I20240502 07:59:47.490119 1682 raft_consensus.cc:348] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } }
I20240502 07:59:47.490243 1682 raft_consensus.cc:729] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66df61031360493886f1170a809e079a, State: Initialized, Role: LEARNER
I20240502 07:59:47.490352 1682 consensus_queue.cc:260] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } }
I20240502 07:59:47.490998 1682 ts_tablet_manager.cc:1398] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:59:47.492062 1077 tablet_copy_service.cc:342] P 77e648af9dbb46fc881340efe1d1e3c3: Request end of tablet copy session 66df61031360493886f1170a809e079a-df4233b549b34b01b6f9662c7e7b3515 received from {username='slave'} at 127.0.0.1:48364
I20240502 07:59:47.492148 1077 tablet_copy_service.cc:433] P 77e648af9dbb46fc881340efe1d1e3c3: ending tablet copy session 66df61031360493886f1170a809e079a-df4233b549b34b01b6f9662c7e7b3515 on tablet df4233b549b34b01b6f9662c7e7b3515 with peer 66df61031360493886f1170a809e079a
I20240502 07:59:47.554657 1605 raft_consensus.cc:1203] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.29->[2.30-2.30] Dedup: 2.30->[]
I20240502 07:59:47.710312 1530 raft_consensus.cc:1203] T c7c1536618ae4e74aa79349d04201fb9 P 66df61031360493886f1170a809e079a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29] Dedup: 1.29->[]
I20240502 07:59:47.715612 1454 raft_consensus.cc:1203] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29] Dedup: 1.29->[]
I20240502 07:59:47.809059 1454 raft_consensus.cc:1203] T fda98cfe14f947969fb41e26c75bc384 P 9d994926b3d54e6395a0d60df8a1db07 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29] Dedup: 1.29->[]
I20240502 07:59:47.918413 1018 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:59:47.919799 1018 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 07:59:47.919909 1018 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:59:47.953287 1704 raft_consensus.cc:1050] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70: attempting to promote NON_VOTER eb0a49d2a951470e85adbcd797128e69 to VOTER
I20240502 07:59:47.953785 1704 consensus_queue.cc:237] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } }
I20240502 07:59:47.955219 1605 raft_consensus.cc:1261] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [term 1 LEARNER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 07:59:47.955293 1218 raft_consensus.cc:1261] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 07:59:47.955650 1063 raft_consensus.cc:1261] T 6edf17550e3f4a0b945477edc6de6af1 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 07:59:47.955646 1704 consensus_queue.cc:1035] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:47.955873 1707 consensus_queue.cc:1035] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:47.956092 1705 consensus_queue.cc:1035] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:47.958268 1704 raft_consensus.cc:2932] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:47.958578 1063 raft_consensus.cc:2932] T 6edf17550e3f4a0b945477edc6de6af1 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:47.960855 1605 raft_consensus.cc:2932] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:47.960953 962 catalog_manager.cc:5484] T 6edf17550e3f4a0b945477edc6de6af1 P 77e648af9dbb46fc881340efe1d1e3c3 reported cstate change: config changed from index 29 to 30, eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b1e9945160714b16b2db91f90c5ccc70" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:47.960969 1218 raft_consensus.cc:2932] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:47.966161 1141 consensus_queue.cc:237] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } }
I20240502 07:59:47.967168 1218 raft_consensus.cc:1261] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240502 07:59:47.967168 1605 raft_consensus.cc:1261] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240502 07:59:47.967557 1707 consensus_queue.cc:1035] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 07:59:47.967761 1704 consensus_queue.cc:1035] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 07:59:47.969281 1705 raft_consensus.cc:2932] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:47.969763 1605 raft_consensus.cc:2932] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:47.970747 1218 raft_consensus.cc:2932] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:47.971057 948 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6edf17550e3f4a0b945477edc6de6af1 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:47.971690 956 catalog_manager.cc:5484] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 reported cstate change: config changed from index 30 to 31, VOTER 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129) evicted. New cstate: current_term: 1 leader_uuid: "b1e9945160714b16b2db91f90c5ccc70" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:47.974495 1056 tablet_service.cc:1514] Processing DeleteTablet for tablet 6edf17550e3f4a0b945477edc6de6af1 with delete_type TABLET_DATA_TOMBSTONED (TS 77e648af9dbb46fc881340efe1d1e3c3 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:48298
I20240502 07:59:47.974887 1715 tablet_replica.cc:330] T 6edf17550e3f4a0b945477edc6de6af1 P 77e648af9dbb46fc881340efe1d1e3c3: stopping tablet replica
I20240502 07:59:47.975107 1715 raft_consensus.cc:2229] T 6edf17550e3f4a0b945477edc6de6af1 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:47.975378 1715 raft_consensus.cc:2258] T 6edf17550e3f4a0b945477edc6de6af1 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:47.975965 1715 ts_tablet_manager.cc:1875] T 6edf17550e3f4a0b945477edc6de6af1 P 77e648af9dbb46fc881340efe1d1e3c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:47.977941 1715 ts_tablet_manager.cc:1888] T 6edf17550e3f4a0b945477edc6de6af1 P 77e648af9dbb46fc881340efe1d1e3c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.30
I20240502 07:59:47.978011 1715 log.cc:1196] T 6edf17550e3f4a0b945477edc6de6af1 P 77e648af9dbb46fc881340efe1d1e3c3: Deleting WAL directory at /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-0-root/wals/6edf17550e3f4a0b945477edc6de6af1
I20240502 07:59:47.978494 949 catalog_manager.cc:4830] TS 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761): tablet 6edf17550e3f4a0b945477edc6de6af1 (table test-workload [id=c4044eda63f741838cbc708a8054d3b9]) successfully deleted
I20240502 07:59:48.008148 1530 raft_consensus.cc:1203] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29] Dedup: 1.29->[]
I20240502 07:59:48.101985 1716 raft_consensus.cc:1050] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f: attempting to promote NON_VOTER eb0a49d2a951470e85adbcd797128e69 to VOTER
I20240502 07:59:48.102571 1716 consensus_queue.cc:237] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 24, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } }
I20240502 07:59:48.104199 1605 raft_consensus.cc:1261] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [term 2 LEARNER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 07:59:48.104283 1141 raft_consensus.cc:1261] T 69a9ebd0c390436b8ace08089f719358 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 07:59:48.104713 1717 consensus_queue.cc:1035] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 07:59:48.105397 1694 consensus_queue.cc:1035] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 07:59:48.107558 1063 raft_consensus.cc:1261] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 07:59:48.108052 1694 consensus_queue.cc:1035] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 07:59:48.110389 1694 raft_consensus.cc:2932] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:48.111052 1063 raft_consensus.cc:2932] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:48.112919 1605 raft_consensus.cc:2932] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:48.113524 960 catalog_manager.cc:5484] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 reported cstate change: config changed from index 30 to 31, eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:48.116469 1141 raft_consensus.cc:2932] T 69a9ebd0c390436b8ace08089f719358 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:48.117126 1218 consensus_queue.cc:237] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 24, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } }
I20240502 07:59:48.118402 1063 raft_consensus.cc:1261] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240502 07:59:48.118466 1605 raft_consensus.cc:1261] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [term 2 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240502 07:59:48.118815 1717 consensus_queue.cc:1035] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 07:59:48.119107 1716 consensus_queue.cc:1035] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 07:59:48.120481 1694 raft_consensus.cc:2932] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:48.122136 951 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 69a9ebd0c390436b8ace08089f719358 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:48.122160 1063 raft_consensus.cc:2932] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:48.123081 1605 raft_consensus.cc:2932] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:48.123781 961 catalog_manager.cc:5484] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f reported cstate change: config changed from index 31 to 32, VOTER b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130) evicted. New cstate: current_term: 2 leader_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:48.126406 1127 tablet_service.cc:1514] Processing DeleteTablet for tablet 69a9ebd0c390436b8ace08089f719358 with delete_type TABLET_DATA_TOMBSTONED (TS b1e9945160714b16b2db91f90c5ccc70 not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:47876
I20240502 07:59:48.126825 1723 tablet_replica.cc:330] T 69a9ebd0c390436b8ace08089f719358 P b1e9945160714b16b2db91f90c5ccc70: stopping tablet replica
I20240502 07:59:48.127022 1723 raft_consensus.cc:2229] T 69a9ebd0c390436b8ace08089f719358 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:48.127213 1723 raft_consensus.cc:2258] T 69a9ebd0c390436b8ace08089f719358 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:48.128348 1723 ts_tablet_manager.cc:1875] T 69a9ebd0c390436b8ace08089f719358 P b1e9945160714b16b2db91f90c5ccc70: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:48.132459 1723 ts_tablet_manager.cc:1888] T 69a9ebd0c390436b8ace08089f719358 P b1e9945160714b16b2db91f90c5ccc70: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.31
I20240502 07:59:48.132567 1723 log.cc:1196] T 69a9ebd0c390436b8ace08089f719358 P b1e9945160714b16b2db91f90c5ccc70: Deleting WAL directory at /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-1-root/wals/69a9ebd0c390436b8ace08089f719358
I20240502 07:59:48.133224 948 catalog_manager.cc:4830] TS b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203): tablet 69a9ebd0c390436b8ace08089f719358 (table test-workload [id=c4044eda63f741838cbc708a8054d3b9]) successfully deleted
I20240502 07:59:48.214088 1704 raft_consensus.cc:1050] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70: attempting to promote NON_VOTER 9d994926b3d54e6395a0d60df8a1db07 to VOTER
I20240502 07:59:48.214623 1704 consensus_queue.cc:237] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } }
I20240502 07:59:48.215212 1690 raft_consensus.cc:1050] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3: attempting to promote NON_VOTER 66df61031360493886f1170a809e079a to VOTER
I20240502 07:59:48.215541 1690 consensus_queue.cc:237] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } }
I20240502 07:59:48.216775 1530 raft_consensus.cc:1261] T c7c1536618ae4e74aa79349d04201fb9 P 66df61031360493886f1170a809e079a [term 1 LEARNER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 07:59:48.217186 1218 raft_consensus.cc:1261] T c7c1536618ae4e74aa79349d04201fb9 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 07:59:48.217463 1066 raft_consensus.cc:1261] T 2c1b6523d6724a02acc9ae19f3f0255e P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 07:59:48.217809 1724 consensus_queue.cc:1035] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:48.218050 1218 raft_consensus.cc:1261] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 07:59:48.218137 1724 consensus_queue.cc:1035] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:48.218397 1139 raft_consensus.cc:1261] T c7c1536618ae4e74aa79349d04201fb9 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 07:59:48.218917 1707 consensus_queue.cc:1035] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:48.219691 1707 consensus_queue.cc:1035] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:48.220109 1454 raft_consensus.cc:1261] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [term 1 LEARNER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 07:59:48.220166 1725 consensus_queue.cc:1035] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:48.220703 1707 consensus_queue.cc:1035] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:48.222509 1707 raft_consensus.cc:2932] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:48.224601 1690 raft_consensus.cc:2932] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 66df61031360493886f1170a809e079a (127.29.106.133) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:48.225347 1454 raft_consensus.cc:2932] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:48.226558 1218 raft_consensus.cc:2932] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:48.227319 1139 raft_consensus.cc:2932] T c7c1536618ae4e74aa79349d04201fb9 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 66df61031360493886f1170a809e079a (127.29.106.133) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:48.227571 1530 raft_consensus.cc:2932] T c7c1536618ae4e74aa79349d04201fb9 P 66df61031360493886f1170a809e079a [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 66df61031360493886f1170a809e079a (127.29.106.133) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:48.228861 1066 raft_consensus.cc:2932] T 2c1b6523d6724a02acc9ae19f3f0255e P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:48.229030 956 catalog_manager.cc:5484] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 reported cstate change: config changed from index 29 to 30, 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b1e9945160714b16b2db91f90c5ccc70" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:48.229945 961 catalog_manager.cc:5484] T c7c1536618ae4e74aa79349d04201fb9 P 66df61031360493886f1170a809e079a reported cstate change: config changed from index 29 to 30, 66df61031360493886f1170a809e079a (127.29.106.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "77e648af9dbb46fc881340efe1d1e3c3" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:48.227334 1211 raft_consensus.cc:2932] T c7c1536618ae4e74aa79349d04201fb9 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 66df61031360493886f1170a809e079a (127.29.106.133) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:48.234150 1139 consensus_queue.cc:237] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } }
I20240502 07:59:48.236331 1454 raft_consensus.cc:1261] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240502 07:59:48.236734 1211 raft_consensus.cc:1261] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240502 07:59:48.237152 1705 consensus_queue.cc:1035] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 07:59:48.237476 1705 consensus_queue.cc:1035] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 07:59:48.238049 1066 consensus_queue.cc:237] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } }
I20240502 07:59:48.239419 1530 raft_consensus.cc:1261] T c7c1536618ae4e74aa79349d04201fb9 P 66df61031360493886f1170a809e079a [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240502 07:59:48.239713 1218 raft_consensus.cc:1261] T c7c1536618ae4e74aa79349d04201fb9 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240502 07:59:48.239876 1725 consensus_queue.cc:1035] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 07:59:48.240024 1690 consensus_queue.cc:1035] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 07:59:48.241287 1687 raft_consensus.cc:2932] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:48.241870 1690 raft_consensus.cc:2932] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:48.241950 1218 raft_consensus.cc:2932] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:48.243012 948 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2c1b6523d6724a02acc9ae19f3f0255e with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:48.243472 949 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c7c1536618ae4e74aa79349d04201fb9 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:48.244901 961 catalog_manager.cc:5484] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f reported cstate change: config changed from index 30 to 31, VOTER 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129) evicted. New cstate: current_term: 1 leader_uuid: "b1e9945160714b16b2db91f90c5ccc70" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:48.244904 1530 raft_consensus.cc:2932] T c7c1536618ae4e74aa79349d04201fb9 P 66df61031360493886f1170a809e079a [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:48.244773 955 catalog_manager.cc:5484] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 reported cstate change: config changed from index 30 to 31, VOTER b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130) evicted. New cstate: current_term: 1 leader_uuid: "77e648af9dbb46fc881340efe1d1e3c3" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:48.247934 1218 raft_consensus.cc:2932] T c7c1536618ae4e74aa79349d04201fb9 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:48.249711 1056 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c1b6523d6724a02acc9ae19f3f0255e with delete_type TABLET_DATA_TOMBSTONED (TS 77e648af9dbb46fc881340efe1d1e3c3 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:48298
I20240502 07:59:48.249926 1715 tablet_replica.cc:330] T 2c1b6523d6724a02acc9ae19f3f0255e P 77e648af9dbb46fc881340efe1d1e3c3: stopping tablet replica
I20240502 07:59:48.250068 1715 raft_consensus.cc:2229] T 2c1b6523d6724a02acc9ae19f3f0255e P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:48.250219 1715 raft_consensus.cc:2258] T 2c1b6523d6724a02acc9ae19f3f0255e P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:48.250797 1127 tablet_service.cc:1514] Processing DeleteTablet for tablet c7c1536618ae4e74aa79349d04201fb9 with delete_type TABLET_DATA_TOMBSTONED (TS b1e9945160714b16b2db91f90c5ccc70 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:47876
I20240502 07:59:48.251063 1723 tablet_replica.cc:330] T c7c1536618ae4e74aa79349d04201fb9 P b1e9945160714b16b2db91f90c5ccc70: stopping tablet replica
I20240502 07:59:48.251224 1723 raft_consensus.cc:2229] T c7c1536618ae4e74aa79349d04201fb9 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:48.251408 1723 raft_consensus.cc:2258] T c7c1536618ae4e74aa79349d04201fb9 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:48.252760 1454 raft_consensus.cc:2932] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:48.253093 1723 ts_tablet_manager.cc:1875] T c7c1536618ae4e74aa79349d04201fb9 P b1e9945160714b16b2db91f90c5ccc70: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:48.255884 1723 ts_tablet_manager.cc:1888] T c7c1536618ae4e74aa79349d04201fb9 P b1e9945160714b16b2db91f90c5ccc70: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.30
I20240502 07:59:48.255999 1723 log.cc:1196] T c7c1536618ae4e74aa79349d04201fb9 P b1e9945160714b16b2db91f90c5ccc70: Deleting WAL directory at /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-1-root/wals/c7c1536618ae4e74aa79349d04201fb9
I20240502 07:59:48.258491 1715 ts_tablet_manager.cc:1875] T 2c1b6523d6724a02acc9ae19f3f0255e P 77e648af9dbb46fc881340efe1d1e3c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:48.259415 948 catalog_manager.cc:4830] TS b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203): tablet c7c1536618ae4e74aa79349d04201fb9 (table test-workload [id=c4044eda63f741838cbc708a8054d3b9]) successfully deleted
I20240502 07:59:48.260975 1715 ts_tablet_manager.cc:1888] T 2c1b6523d6724a02acc9ae19f3f0255e P 77e648af9dbb46fc881340efe1d1e3c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.30
I20240502 07:59:48.261058 1715 log.cc:1196] T 2c1b6523d6724a02acc9ae19f3f0255e P 77e648af9dbb46fc881340efe1d1e3c3: Deleting WAL directory at /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-0-root/wals/2c1b6523d6724a02acc9ae19f3f0255e
I20240502 07:59:48.261606 949 catalog_manager.cc:4830] TS 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761): tablet 2c1b6523d6724a02acc9ae19f3f0255e (table test-workload [id=c4044eda63f741838cbc708a8054d3b9]) successfully deleted
I20240502 07:59:48.274471 1725 raft_consensus.cc:1050] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3: attempting to promote NON_VOTER 9d994926b3d54e6395a0d60df8a1db07 to VOTER
I20240502 07:59:48.274899 1725 consensus_queue.cc:237] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } }
I20240502 07:59:48.276310 1218 raft_consensus.cc:1261] T fda98cfe14f947969fb41e26c75bc384 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 07:59:48.276412 1139 raft_consensus.cc:1261] T fda98cfe14f947969fb41e26c75bc384 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 07:59:48.276345 1454 raft_consensus.cc:1261] T fda98cfe14f947969fb41e26c75bc384 P 9d994926b3d54e6395a0d60df8a1db07 [term 1 LEARNER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 07:59:48.276872 1724 consensus_queue.cc:1035] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:48.277281 1724 consensus_queue.cc:1035] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:48.277616 1724 consensus_queue.cc:1035] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:48.280010 1690 raft_consensus.cc:2932] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:48.280486 1454 raft_consensus.cc:2932] T fda98cfe14f947969fb41e26c75bc384 P 9d994926b3d54e6395a0d60df8a1db07 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:48.282083 1143 raft_consensus.cc:2932] T fda98cfe14f947969fb41e26c75bc384 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:48.283160 1218 raft_consensus.cc:2932] T fda98cfe14f947969fb41e26c75bc384 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:48.284881 962 catalog_manager.cc:5484] T fda98cfe14f947969fb41e26c75bc384 P 9d994926b3d54e6395a0d60df8a1db07 reported cstate change: config changed from index 29 to 30, 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "77e648af9dbb46fc881340efe1d1e3c3" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:48.289585 1066 consensus_queue.cc:237] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } }
I20240502 07:59:48.291113 1143 raft_consensus.cc:1261] T fda98cfe14f947969fb41e26c75bc384 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240502 07:59:48.291113 1454 raft_consensus.cc:1261] T fda98cfe14f947969fb41e26c75bc384 P 9d994926b3d54e6395a0d60df8a1db07 [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240502 07:59:48.291561 1724 consensus_queue.cc:1035] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 07:59:48.291920 1724 consensus_queue.cc:1035] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 07:59:48.293318 1724 raft_consensus.cc:2932] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:48.295184 949 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fda98cfe14f947969fb41e26c75bc384 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:48.295593 1143 raft_consensus.cc:2932] T fda98cfe14f947969fb41e26c75bc384 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:48.295671 1454 raft_consensus.cc:2932] T fda98cfe14f947969fb41e26c75bc384 P 9d994926b3d54e6395a0d60df8a1db07 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:48.296129 961 catalog_manager.cc:5484] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 reported cstate change: config changed from index 30 to 31, VOTER 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131) evicted. New cstate: current_term: 1 leader_uuid: "77e648af9dbb46fc881340efe1d1e3c3" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:48.299433 1207 tablet_service.cc:1514] Processing DeleteTablet for tablet fda98cfe14f947969fb41e26c75bc384 with delete_type TABLET_DATA_TOMBSTONED (TS 27ec1b26c2f64cc682ed7ae12e133a9f not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:52036
I20240502 07:59:48.299876 1741 tablet_replica.cc:330] T fda98cfe14f947969fb41e26c75bc384 P 27ec1b26c2f64cc682ed7ae12e133a9f: stopping tablet replica
I20240502 07:59:48.300076 1741 raft_consensus.cc:2229] T fda98cfe14f947969fb41e26c75bc384 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:48.300575 1741 raft_consensus.cc:2258] T fda98cfe14f947969fb41e26c75bc384 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:48.301141 1741 ts_tablet_manager.cc:1875] T fda98cfe14f947969fb41e26c75bc384 P 27ec1b26c2f64cc682ed7ae12e133a9f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:48.303579 1741 ts_tablet_manager.cc:1888] T fda98cfe14f947969fb41e26c75bc384 P 27ec1b26c2f64cc682ed7ae12e133a9f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.30
I20240502 07:59:48.303668 1741 log.cc:1196] T fda98cfe14f947969fb41e26c75bc384 P 27ec1b26c2f64cc682ed7ae12e133a9f: Deleting WAL directory at /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-2-root/wals/fda98cfe14f947969fb41e26c75bc384
I20240502 07:59:48.304230 951 catalog_manager.cc:4830] TS 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737): tablet fda98cfe14f947969fb41e26c75bc384 (table test-workload [id=c4044eda63f741838cbc708a8054d3b9]) successfully deleted
I20240502 07:59:48.393391 1737 raft_consensus.cc:1050] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3: attempting to promote NON_VOTER 66df61031360493886f1170a809e079a to VOTER
I20240502 07:59:48.393949 1737 consensus_queue.cc:237] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } }
I20240502 07:59:48.395648 1530 raft_consensus.cc:1261] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [term 1 LEARNER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 07:59:48.395648 1143 raft_consensus.cc:1261] T df4233b549b34b01b6f9662c7e7b3515 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 07:59:48.395741 1218 raft_consensus.cc:1261] T df4233b549b34b01b6f9662c7e7b3515 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 07:59:48.396310 1724 consensus_queue.cc:1035] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:48.396477 1690 consensus_queue.cc:1035] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:48.396744 1725 consensus_queue.cc:1035] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:48.399225 1724 raft_consensus.cc:2932] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 66df61031360493886f1170a809e079a (127.29.106.133) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:48.399909 1143 raft_consensus.cc:2932] T df4233b549b34b01b6f9662c7e7b3515 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 66df61031360493886f1170a809e079a (127.29.106.133) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:48.400187 1530 raft_consensus.cc:2932] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 66df61031360493886f1170a809e079a (127.29.106.133) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:48.402201 960 catalog_manager.cc:5484] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a reported cstate change: config changed from index 29 to 30, 66df61031360493886f1170a809e079a (127.29.106.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "77e648af9dbb46fc881340efe1d1e3c3" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:48.402519 1218 raft_consensus.cc:2932] T df4233b549b34b01b6f9662c7e7b3515 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 66df61031360493886f1170a809e079a (127.29.106.133) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:48.405889 1066 consensus_queue.cc:237] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } }
I20240502 07:59:48.406899 1143 raft_consensus.cc:1261] T df4233b549b34b01b6f9662c7e7b3515 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240502 07:59:48.407209 1737 consensus_queue.cc:1035] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 07:59:48.407549 1530 raft_consensus.cc:1261] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240502 07:59:48.408178 1737 consensus_queue.cc:1035] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 07:59:48.409008 1737 raft_consensus.cc:2932] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:48.410522 1530 raft_consensus.cc:2932] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:48.410732 949 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet df4233b549b34b01b6f9662c7e7b3515 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:48.411689 961 catalog_manager.cc:5484] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 reported cstate change: config changed from index 30 to 31, VOTER 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131) evicted. New cstate: current_term: 1 leader_uuid: "77e648af9dbb46fc881340efe1d1e3c3" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:48.413197 1143 raft_consensus.cc:2932] T df4233b549b34b01b6f9662c7e7b3515 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:48.414640 1207 tablet_service.cc:1514] Processing DeleteTablet for tablet df4233b549b34b01b6f9662c7e7b3515 with delete_type TABLET_DATA_TOMBSTONED (TS 27ec1b26c2f64cc682ed7ae12e133a9f not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:52036
I20240502 07:59:48.414851 1741 tablet_replica.cc:330] T df4233b549b34b01b6f9662c7e7b3515 P 27ec1b26c2f64cc682ed7ae12e133a9f: stopping tablet replica
I20240502 07:59:48.414999 1741 raft_consensus.cc:2229] T df4233b549b34b01b6f9662c7e7b3515 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:48.415149 1741 raft_consensus.cc:2258] T df4233b549b34b01b6f9662c7e7b3515 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:48.415774 1741 ts_tablet_manager.cc:1875] T df4233b549b34b01b6f9662c7e7b3515 P 27ec1b26c2f64cc682ed7ae12e133a9f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:48.417742 1741 ts_tablet_manager.cc:1888] T df4233b549b34b01b6f9662c7e7b3515 P 27ec1b26c2f64cc682ed7ae12e133a9f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.30
I20240502 07:59:48.417809 1741 log.cc:1196] T df4233b549b34b01b6f9662c7e7b3515 P 27ec1b26c2f64cc682ed7ae12e133a9f: Deleting WAL directory at /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-2-root/wals/df4233b549b34b01b6f9662c7e7b3515
I20240502 07:59:48.418239 951 catalog_manager.cc:4830] TS 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737): tablet df4233b549b34b01b6f9662c7e7b3515 (table test-workload [id=c4044eda63f741838cbc708a8054d3b9]) successfully deleted
I20240502 07:59:49.414973 1066 consensus_queue.cc:237] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } }
I20240502 07:59:49.416478 1530 raft_consensus.cc:1261] T c7c1536618ae4e74aa79349d04201fb9 P 66df61031360493886f1170a809e079a [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
W20240502 07:59:49.416772 1030 consensus_peers.cc:485] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 -> Peer 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132:44697): Couldn't send request to peer 9d994926b3d54e6395a0d60df8a1db07. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c7c1536618ae4e74aa79349d04201fb9. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:49.416862 1218 raft_consensus.cc:1261] T c7c1536618ae4e74aa79349d04201fb9 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20240502 07:59:49.416926 1725 consensus_queue.cc:1035] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 07:59:49.417209 1749 consensus_queue.cc:1035] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 07:59:49.418859 1748 raft_consensus.cc:2932] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Committing config change with OpId 1.32: config changed from index 31 to 32, NON_VOTER 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } } }
I20240502 07:59:49.419209 1218 raft_consensus.cc:2932] T c7c1536618ae4e74aa79349d04201fb9 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, NON_VOTER 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } } }
I20240502 07:59:49.419253 1530 raft_consensus.cc:2932] T c7c1536618ae4e74aa79349d04201fb9 P 66df61031360493886f1170a809e079a [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, NON_VOTER 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } } }
I20240502 07:59:49.421375 961 catalog_manager.cc:5484] T c7c1536618ae4e74aa79349d04201fb9 P 66df61031360493886f1170a809e079a reported cstate change: config changed from index 31 to 32, NON_VOTER 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) added. New cstate: current_term: 1 leader_uuid: "77e648af9dbb46fc881340efe1d1e3c3" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } } }
I20240502 07:59:49.421496 1218 consensus_queue.cc:237] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 22, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } }
I20240502 07:59:49.423085 1143 raft_consensus.cc:1261] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 07:59:49.423384 1066 raft_consensus.cc:1261] T 2ecebe8dad3946bd956afacf695fabaa P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 07:59:49.423924 1746 consensus_queue.cc:1035] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 07:59:49.424466 1717 consensus_queue.cc:1035] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 07:59:49.426234 1746 raft_consensus.cc:2932] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER 66df61031360493886f1170a809e079a (127.29.106.133) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } } }
I20240502 07:59:49.427243 1066 raft_consensus.cc:2932] T 2ecebe8dad3946bd956afacf695fabaa P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER 66df61031360493886f1170a809e079a (127.29.106.133) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } } }
W20240502 07:59:49.428392 1184 consensus_peers.cc:485] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f -> Peer 66df61031360493886f1170a809e079a (127.29.106.133:38717): Couldn't send request to peer 66df61031360493886f1170a809e079a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ecebe8dad3946bd956afacf695fabaa. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:49.427263 1143 raft_consensus.cc:2932] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER 66df61031360493886f1170a809e079a (127.29.106.133) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } } }
I20240502 07:59:49.429473 1066 consensus_queue.cc:237] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } }
I20240502 07:59:49.429606 960 catalog_manager.cc:5484] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f reported cstate change: config changed from index -1 to 28, NON_VOTER 66df61031360493886f1170a809e079a (127.29.106.133) added. New cstate: current_term: 2 leader_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:49.431221 1143 raft_consensus.cc:1261] T fda98cfe14f947969fb41e26c75bc384 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20240502 07:59:49.431221 1454 raft_consensus.cc:1261] T fda98cfe14f947969fb41e26c75bc384 P 9d994926b3d54e6395a0d60df8a1db07 [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20240502 07:59:49.431660 1749 consensus_queue.cc:1035] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 07:59:49.432202 1749 consensus_queue.cc:1035] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 07:59:49.433749 1749 raft_consensus.cc:2932] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Committing config change with OpId 1.32: config changed from index 31 to 32, NON_VOTER eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } } }
I20240502 07:59:49.434405 1454 raft_consensus.cc:2932] T fda98cfe14f947969fb41e26c75bc384 P 9d994926b3d54e6395a0d60df8a1db07 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, NON_VOTER eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } } }
I20240502 07:59:49.436745 1143 raft_consensus.cc:2932] T fda98cfe14f947969fb41e26c75bc384 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, NON_VOTER eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } } }
I20240502 07:59:49.437405 955 catalog_manager.cc:5484] T fda98cfe14f947969fb41e26c75bc384 P 9d994926b3d54e6395a0d60df8a1db07 reported cstate change: config changed from index 31 to 32, NON_VOTER eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) added. New cstate: current_term: 1 leader_uuid: "77e648af9dbb46fc881340efe1d1e3c3" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } } }
I20240502 07:59:49.438237 1139 consensus_queue.cc:237] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } }
W20240502 07:59:49.438316 1033 consensus_peers.cc:485] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 -> Peer eb0a49d2a951470e85adbcd797128e69 (127.29.106.134:34541): Couldn't send request to peer eb0a49d2a951470e85adbcd797128e69. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fda98cfe14f947969fb41e26c75bc384. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:49.439488 1066 raft_consensus.cc:1261] T 5437f94679a841e3980053e69d624d76 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 07:59:49.440179 1218 raft_consensus.cc:1261] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 07:59:49.440222 1772 consensus_queue.cc:1035] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
W20240502 07:59:49.440780 1109 consensus_peers.cc:485] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 -> Peer eb0a49d2a951470e85adbcd797128e69 (127.29.106.134:34541): Couldn't send request to peer eb0a49d2a951470e85adbcd797128e69. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5437f94679a841e3980053e69d624d76. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:49.440991 1772 consensus_queue.cc:1035] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 07:59:49.441932 1771 raft_consensus.cc:2932] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } } }
I20240502 07:59:49.442312 1066 raft_consensus.cc:2932] T 5437f94679a841e3980053e69d624d76 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } } }
I20240502 07:59:49.442931 1218 raft_consensus.cc:2932] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } } }
I20240502 07:59:49.444430 960 catalog_manager.cc:5484] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 reported cstate change: config changed from index -1 to 29, NON_VOTER eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) added. New cstate: current_term: 1 leader_uuid: "b1e9945160714b16b2db91f90c5ccc70" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:49.444895 1139 consensus_queue.cc:237] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } }
I20240502 07:59:49.446434 1218 raft_consensus.cc:1261] T cdf0ff2967c040a589f915172d3f446b P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
W20240502 07:59:49.446787 1106 consensus_peers.cc:485] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 -> Peer 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132:44697): Couldn't send request to peer 9d994926b3d54e6395a0d60df8a1db07. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cdf0ff2967c040a589f915172d3f446b. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:49.446841 1066 raft_consensus.cc:1261] T cdf0ff2967c040a589f915172d3f446b P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 07:59:49.447219 1771 consensus_queue.cc:1035] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 07:59:49.447427 1734 consensus_queue.cc:1035] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 07:59:49.448908 1771 raft_consensus.cc:2932] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } } }
I20240502 07:59:49.449733 1218 raft_consensus.cc:2932] T cdf0ff2967c040a589f915172d3f446b P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } } }
I20240502 07:59:49.450451 1066 raft_consensus.cc:2932] T cdf0ff2967c040a589f915172d3f446b P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } } }
I20240502 07:59:49.451148 961 catalog_manager.cc:5484] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 reported cstate change: config changed from index -1 to 28, NON_VOTER 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) added. New cstate: current_term: 1 leader_uuid: "b1e9945160714b16b2db91f90c5ccc70" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:49.451361 1211 consensus_queue.cc:237] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } }
I20240502 07:59:49.453071 1139 raft_consensus.cc:1261] T 58c5f52a420c467684b2cb50972e7c99 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
W20240502 07:59:49.453307 1184 consensus_peers.cc:485] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f -> Peer 66df61031360493886f1170a809e079a (127.29.106.133:38717): Couldn't send request to peer 66df61031360493886f1170a809e079a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 58c5f52a420c467684b2cb50972e7c99. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:49.453435 1717 consensus_queue.cc:1035] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 07:59:49.454545 1066 raft_consensus.cc:1261] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 07:59:49.455039 1717 consensus_queue.cc:1035] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 07:59:49.456148 1717 raft_consensus.cc:2932] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 66df61031360493886f1170a809e079a (127.29.106.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } } }
I20240502 07:59:49.457319 1066 raft_consensus.cc:2932] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 66df61031360493886f1170a809e079a (127.29.106.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } } }
I20240502 07:59:49.458523 1139 raft_consensus.cc:2932] T 58c5f52a420c467684b2cb50972e7c99 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 66df61031360493886f1170a809e079a (127.29.106.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } } }
I20240502 07:59:49.458935 960 catalog_manager.cc:5484] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f reported cstate change: config changed from index -1 to 29, NON_VOTER 66df61031360493886f1170a809e079a (127.29.106.133) added. New cstate: current_term: 1 leader_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:49.855191 1782 ts_tablet_manager.cc:902] T 2ecebe8dad3946bd956afacf695fabaa P 66df61031360493886f1170a809e079a: Initiating tablet copy from peer 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:49.855737 1782 tablet_copy_client.cc:308] T 2ecebe8dad3946bd956afacf695fabaa P 66df61031360493886f1170a809e079a: tablet copy: Beginning tablet copy session from remote peer at address 127.29.106.131:36737
I20240502 07:59:49.860572 1228 tablet_copy_service.cc:140] P 27ec1b26c2f64cc682ed7ae12e133a9f: Received BeginTabletCopySession request for tablet 2ecebe8dad3946bd956afacf695fabaa from peer 66df61031360493886f1170a809e079a ({username='slave'} at 127.0.0.1:40304)
I20240502 07:59:49.860702 1228 tablet_copy_service.cc:161] P 27ec1b26c2f64cc682ed7ae12e133a9f: Beginning new tablet copy session on tablet 2ecebe8dad3946bd956afacf695fabaa from peer 66df61031360493886f1170a809e079a at {username='slave'} at 127.0.0.1:40304: session id = 66df61031360493886f1170a809e079a-2ecebe8dad3946bd956afacf695fabaa
I20240502 07:59:49.861799 1228 tablet_copy_source_session.cc:205] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:49.862546 1782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ecebe8dad3946bd956afacf695fabaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:49.864778 1782 tablet_copy_client.cc:783] T 2ecebe8dad3946bd956afacf695fabaa P 66df61031360493886f1170a809e079a: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:49.864933 1782 tablet_copy_client.cc:647] T 2ecebe8dad3946bd956afacf695fabaa P 66df61031360493886f1170a809e079a: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:49.866209 1786 tablet_copy_client.cc:1002] T 2ecebe8dad3946bd956afacf695fabaa P 66df61031360493886f1170a809e079a: tablet copy: Injecting latency into file download: 2000
I20240502 07:59:49.882606 1789 ts_tablet_manager.cc:902] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07: Initiating tablet copy from peer b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203)
I20240502 07:59:49.883246 1789 tablet_copy_client.cc:308] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07: tablet copy: Beginning tablet copy session from remote peer at address 127.29.106.130:38203
I20240502 07:59:49.883896 1153 tablet_copy_service.cc:140] P b1e9945160714b16b2db91f90c5ccc70: Received BeginTabletCopySession request for tablet cdf0ff2967c040a589f915172d3f446b from peer 9d994926b3d54e6395a0d60df8a1db07 ({username='slave'} at 127.0.0.1:47922)
I20240502 07:59:49.884009 1153 tablet_copy_service.cc:161] P b1e9945160714b16b2db91f90c5ccc70: Beginning new tablet copy session on tablet cdf0ff2967c040a589f915172d3f446b from peer 9d994926b3d54e6395a0d60df8a1db07 at {username='slave'} at 127.0.0.1:47922: session id = 9d994926b3d54e6395a0d60df8a1db07-cdf0ff2967c040a589f915172d3f446b
I20240502 07:59:49.885308 1153 tablet_copy_source_session.cc:205] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:49.886370 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdf0ff2967c040a589f915172d3f446b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:49.888412 1789 tablet_copy_client.cc:783] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:49.888572 1789 tablet_copy_client.cc:647] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:49.889611 1792 tablet_copy_client.cc:1002] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07: tablet copy: Injecting latency into file download: 2000
I20240502 07:59:49.920158 1018 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:59:49.922114 1139 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2c1b6523d6724a02acc9ae19f3f0255e"
dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
mode: GRACEFUL
new_leader_uuid: "9d994926b3d54e6395a0d60df8a1db07"
from {username='slave'} at 127.0.0.1:47914
I20240502 07:59:49.922246 1139 raft_consensus.cc:595] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Received request to transfer leadership to TS 9d994926b3d54e6395a0d60df8a1db07
I20240502 07:59:49.931892 1218 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2ecebe8dad3946bd956afacf695fabaa"
dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
mode: GRACEFUL
new_leader_uuid: "b1e9945160714b16b2db91f90c5ccc70"
from {username='slave'} at 127.0.0.1:40314
I20240502 07:59:49.932060 1218 raft_consensus.cc:595] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Received request to transfer leadership to TS b1e9945160714b16b2db91f90c5ccc70
I20240502 07:59:49.932747 1218 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "69a9ebd0c390436b8ace08089f719358"
dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
mode: GRACEFUL
new_leader_uuid: "eb0a49d2a951470e85adbcd797128e69"
from {username='slave'} at 127.0.0.1:40314
I20240502 07:59:49.932840 1218 raft_consensus.cc:595] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Received request to transfer leadership to TS eb0a49d2a951470e85adbcd797128e69
I20240502 07:59:49.933490 1139 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6edf17550e3f4a0b945477edc6de6af1"
dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
mode: GRACEFUL
new_leader_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
from {username='slave'} at 127.0.0.1:47914
I20240502 07:59:49.933583 1139 raft_consensus.cc:595] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Received request to transfer leadership to TS 27ec1b26c2f64cc682ed7ae12e133a9f
I20240502 07:59:49.934454 1063 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0d982dd52d246348fb3984fd1c3a40f"
dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3"
mode: GRACEFUL
new_leader_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
from {username='slave'} at 127.0.0.1:48332
I20240502 07:59:49.934644 1063 raft_consensus.cc:595] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Received request to transfer leadership to TS 27ec1b26c2f64cc682ed7ae12e133a9f
I20240502 07:59:49.935204 1795 ts_tablet_manager.cc:902] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a: Initiating tablet copy from peer 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:49.935458 1063 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df4233b549b34b01b6f9662c7e7b3515"
dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3"
mode: GRACEFUL
new_leader_uuid: "66df61031360493886f1170a809e079a"
from {username='slave'} at 127.0.0.1:48332
I20240502 07:59:49.935623 1063 raft_consensus.cc:595] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Received request to transfer leadership to TS 66df61031360493886f1170a809e079a
I20240502 07:59:49.935793 1795 tablet_copy_client.cc:308] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a: tablet copy: Beginning tablet copy session from remote peer at address 127.29.106.131:36737
I20240502 07:59:49.935969 1018 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 6
I20240502 07:59:49.936105 1018 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:59:49.936254 1228 tablet_copy_service.cc:140] P 27ec1b26c2f64cc682ed7ae12e133a9f: Received BeginTabletCopySession request for tablet 58c5f52a420c467684b2cb50972e7c99 from peer 66df61031360493886f1170a809e079a ({username='slave'} at 127.0.0.1:40304)
I20240502 07:59:49.936383 1228 tablet_copy_service.cc:161] P 27ec1b26c2f64cc682ed7ae12e133a9f: Beginning new tablet copy session on tablet 58c5f52a420c467684b2cb50972e7c99 from peer 66df61031360493886f1170a809e079a at {username='slave'} at 127.0.0.1:40304: session id = 66df61031360493886f1170a809e079a-58c5f52a420c467684b2cb50972e7c99
I20240502 07:59:49.937470 1228 tablet_copy_source_session.cc:205] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:49.938305 1795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58c5f52a420c467684b2cb50972e7c99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:49.940567 1795 tablet_copy_client.cc:783] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:49.940758 1795 tablet_copy_client.cc:647] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:49.941701 1799 tablet_copy_client.cc:1002] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a: tablet copy: Injecting latency into file download: 2000
I20240502 07:59:49.942193 1772 raft_consensus.cc:979] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70: : Instructing follower 27ec1b26c2f64cc682ed7ae12e133a9f to start an election
I20240502 07:59:49.942293 1772 raft_consensus.cc:1067] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Signalling peer 27ec1b26c2f64cc682ed7ae12e133a9f to start an election
I20240502 07:59:49.942670 1218 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6edf17550e3f4a0b945477edc6de6af1"
dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
from {username='slave'} at 127.0.0.1:52044
I20240502 07:59:49.942799 1218 raft_consensus.cc:482] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:49.942867 1218 raft_consensus.cc:3037] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:49.943733 1218 raft_consensus.cc:504] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } }
I20240502 07:59:49.944098 1218 leader_election.cc:288] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [CANDIDATE]: Term 2 election: Requested vote from peers b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203), eb0a49d2a951470e85adbcd797128e69 (127.29.106.134:34541)
I20240502 07:59:49.944370 1139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6edf17550e3f4a0b945477edc6de6af1" candidate_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
I20240502 07:59:49.944476 1139 raft_consensus.cc:3032] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:49.944535 1139 raft_consensus.cc:729] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b1e9945160714b16b2db91f90c5ccc70, State: Running, Role: LEADER
I20240502 07:59:49.944577 1605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6edf17550e3f4a0b945477edc6de6af1" candidate_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "eb0a49d2a951470e85adbcd797128e69"
I20240502 07:59:49.944690 1605 raft_consensus.cc:3037] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:49.944685 1139 consensus_queue.cc:260] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } }
I20240502 07:59:49.944864 1139 raft_consensus.cc:3037] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:49.945647 1605 raft_consensus.cc:2454] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27ec1b26c2f64cc682ed7ae12e133a9f in term 2.
I20240502 07:59:49.945816 1139 raft_consensus.cc:2454] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27ec1b26c2f64cc682ed7ae12e133a9f in term 2.
I20240502 07:59:49.946072 1184 leader_election.cc:302] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ec1b26c2f64cc682ed7ae12e133a9f, eb0a49d2a951470e85adbcd797128e69; no voters:
I20240502 07:59:49.946218 1746 raft_consensus.cc:2781] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:49.946309 1746 raft_consensus.cc:686] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Becoming Leader. State: Replica: 27ec1b26c2f64cc682ed7ae12e133a9f, State: Running, Role: LEADER
I20240502 07:59:49.946436 1746 consensus_queue.cc:237] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } }
I20240502 07:59:49.947913 961 catalog_manager.cc:5484] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f reported cstate change: term changed from 1 to 2, leader changed from b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130) to 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131). New cstate: current_term: 2 leader_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:49.972829 1801 ts_tablet_manager.cc:902] T fda98cfe14f947969fb41e26c75bc384 P eb0a49d2a951470e85adbcd797128e69: Initiating tablet copy from peer 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761)
I20240502 07:59:49.973393 1801 tablet_copy_client.cc:308] T fda98cfe14f947969fb41e26c75bc384 P eb0a49d2a951470e85adbcd797128e69: tablet copy: Beginning tablet copy session from remote peer at address 127.29.106.129:34761
I20240502 07:59:49.977329 1077 tablet_copy_service.cc:140] P 77e648af9dbb46fc881340efe1d1e3c3: Received BeginTabletCopySession request for tablet fda98cfe14f947969fb41e26c75bc384 from peer eb0a49d2a951470e85adbcd797128e69 ({username='slave'} at 127.0.0.1:54320)
I20240502 07:59:49.977450 1077 tablet_copy_service.cc:161] P 77e648af9dbb46fc881340efe1d1e3c3: Beginning new tablet copy session on tablet fda98cfe14f947969fb41e26c75bc384 from peer eb0a49d2a951470e85adbcd797128e69 at {username='slave'} at 127.0.0.1:54320: session id = eb0a49d2a951470e85adbcd797128e69-fda98cfe14f947969fb41e26c75bc384
I20240502 07:59:49.978495 1077 tablet_copy_source_session.cc:205] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:49.979261 1801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fda98cfe14f947969fb41e26c75bc384. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:49.981498 1801 tablet_copy_client.cc:783] T fda98cfe14f947969fb41e26c75bc384 P eb0a49d2a951470e85adbcd797128e69: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:49.981667 1801 tablet_copy_client.cc:647] T fda98cfe14f947969fb41e26c75bc384 P eb0a49d2a951470e85adbcd797128e69: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:49.982582 1805 tablet_copy_client.cc:1002] T fda98cfe14f947969fb41e26c75bc384 P eb0a49d2a951470e85adbcd797128e69: tablet copy: Injecting latency into file download: 2000
I20240502 07:59:49.987303 1725 raft_consensus.cc:979] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3: : Instructing follower 27ec1b26c2f64cc682ed7ae12e133a9f to start an election
I20240502 07:59:49.987393 1725 raft_consensus.cc:1067] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Signalling peer 27ec1b26c2f64cc682ed7ae12e133a9f to start an election
I20240502 07:59:49.987798 1218 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d0d982dd52d246348fb3984fd1c3a40f"
dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
from {username='slave'} at 127.0.0.1:52060
I20240502 07:59:49.987927 1218 raft_consensus.cc:482] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:49.987995 1218 raft_consensus.cc:3037] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:49.988962 1218 raft_consensus.cc:504] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:49.989310 1218 leader_election.cc:288] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [CANDIDATE]: Term 2 election: Requested vote from peers 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761), b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203)
I20240502 07:59:49.989658 1139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d982dd52d246348fb3984fd1c3a40f" candidate_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
I20240502 07:59:49.989696 1063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d982dd52d246348fb3984fd1c3a40f" candidate_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3"
I20240502 07:59:49.989771 1139 raft_consensus.cc:3037] T d0d982dd52d246348fb3984fd1c3a40f P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:49.989794 1063 raft_consensus.cc:3032] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:49.989835 1063 raft_consensus.cc:729] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77e648af9dbb46fc881340efe1d1e3c3, State: Running, Role: LEADER
I20240502 07:59:49.989965 1063 consensus_queue.cc:260] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:49.990100 1063 raft_consensus.cc:3037] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:49.990654 1139 raft_consensus.cc:2454] T d0d982dd52d246348fb3984fd1c3a40f P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27ec1b26c2f64cc682ed7ae12e133a9f in term 2.
I20240502 07:59:49.990921 1181 leader_election.cc:302] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ec1b26c2f64cc682ed7ae12e133a9f, b1e9945160714b16b2db91f90c5ccc70; no voters:
I20240502 07:59:49.991015 1063 raft_consensus.cc:2454] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27ec1b26c2f64cc682ed7ae12e133a9f in term 2.
I20240502 07:59:49.991065 1746 raft_consensus.cc:2781] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:49.991135 1746 raft_consensus.cc:686] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Becoming Leader. State: Replica: 27ec1b26c2f64cc682ed7ae12e133a9f, State: Running, Role: LEADER
I20240502 07:59:49.991282 1746 consensus_queue.cc:237] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:49.991412 1809 ts_tablet_manager.cc:902] T 5437f94679a841e3980053e69d624d76 P eb0a49d2a951470e85adbcd797128e69: Initiating tablet copy from peer b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203)
I20240502 07:59:49.991973 1809 tablet_copy_client.cc:308] T 5437f94679a841e3980053e69d624d76 P eb0a49d2a951470e85adbcd797128e69: tablet copy: Beginning tablet copy session from remote peer at address 127.29.106.130:38203
I20240502 07:59:49.992892 1153 tablet_copy_service.cc:140] P b1e9945160714b16b2db91f90c5ccc70: Received BeginTabletCopySession request for tablet 5437f94679a841e3980053e69d624d76 from peer eb0a49d2a951470e85adbcd797128e69 ({username='slave'} at 127.0.0.1:47924)
I20240502 07:59:49.992988 1153 tablet_copy_service.cc:161] P b1e9945160714b16b2db91f90c5ccc70: Beginning new tablet copy session on tablet 5437f94679a841e3980053e69d624d76 from peer eb0a49d2a951470e85adbcd797128e69 at {username='slave'} at 127.0.0.1:47924: session id = eb0a49d2a951470e85adbcd797128e69-5437f94679a841e3980053e69d624d76
I20240502 07:59:49.993487 961 catalog_manager.cc:5484] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f reported cstate change: term changed from 1 to 2, leader changed from 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129) to 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131). New cstate: current_term: 2 leader_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: HEALTHY } } }
I20240502 07:59:49.993702 1153 tablet_copy_source_session.cc:205] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:49.994957 1809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5437f94679a841e3980053e69d624d76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:49.997110 1809 tablet_copy_client.cc:783] T 5437f94679a841e3980053e69d624d76 P eb0a49d2a951470e85adbcd797128e69: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:49.997264 1809 tablet_copy_client.cc:647] T 5437f94679a841e3980053e69d624d76 P eb0a49d2a951470e85adbcd797128e69: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:49.998507 1810 tablet_copy_client.cc:1002] T 5437f94679a841e3980053e69d624d76 P eb0a49d2a951470e85adbcd797128e69: tablet copy: Injecting latency into file download: 2000
I20240502 07:59:50.002259 1815 ts_tablet_manager.cc:902] T c7c1536618ae4e74aa79349d04201fb9 P 9d994926b3d54e6395a0d60df8a1db07: Initiating tablet copy from peer 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761)
I20240502 07:59:50.002858 1815 tablet_copy_client.cc:308] T c7c1536618ae4e74aa79349d04201fb9 P 9d994926b3d54e6395a0d60df8a1db07: tablet copy: Beginning tablet copy session from remote peer at address 127.29.106.129:34761
I20240502 07:59:50.003383 1077 tablet_copy_service.cc:140] P 77e648af9dbb46fc881340efe1d1e3c3: Received BeginTabletCopySession request for tablet c7c1536618ae4e74aa79349d04201fb9 from peer 9d994926b3d54e6395a0d60df8a1db07 ({username='slave'} at 127.0.0.1:48350)
I20240502 07:59:50.003521 1077 tablet_copy_service.cc:161] P 77e648af9dbb46fc881340efe1d1e3c3: Beginning new tablet copy session on tablet c7c1536618ae4e74aa79349d04201fb9 from peer 9d994926b3d54e6395a0d60df8a1db07 at {username='slave'} at 127.0.0.1:48350: session id = 9d994926b3d54e6395a0d60df8a1db07-c7c1536618ae4e74aa79349d04201fb9
I20240502 07:59:50.004592 1077 tablet_copy_source_session.cc:205] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:50.005394 1815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7c1536618ae4e74aa79349d04201fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:50.007400 1815 tablet_copy_client.cc:783] T c7c1536618ae4e74aa79349d04201fb9 P 9d994926b3d54e6395a0d60df8a1db07: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:50.007647 1815 tablet_copy_client.cc:647] T c7c1536618ae4e74aa79349d04201fb9 P 9d994926b3d54e6395a0d60df8a1db07: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:50.008773 1819 tablet_copy_client.cc:1002] T c7c1536618ae4e74aa79349d04201fb9 P 9d994926b3d54e6395a0d60df8a1db07: tablet copy: Injecting latency into file download: 2000
I20240502 07:59:50.163851 1746 raft_consensus.cc:979] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f: : Instructing follower eb0a49d2a951470e85adbcd797128e69 to start an election
I20240502 07:59:50.163976 1746 raft_consensus.cc:1067] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Signalling peer eb0a49d2a951470e85adbcd797128e69 to start an election
I20240502 07:59:50.164440 1605 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "69a9ebd0c390436b8ace08089f719358"
dest_uuid: "eb0a49d2a951470e85adbcd797128e69"
from {username='slave'} at 127.0.0.1:35580
I20240502 07:59:50.164609 1605 raft_consensus.cc:482] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:50.164690 1605 raft_consensus.cc:3037] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [term 2 FOLLOWER]: Advancing to term 3
I20240502 07:59:50.165552 1605 raft_consensus.cc:504] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } }
I20240502 07:59:50.166050 1605 leader_election.cc:288] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [CANDIDATE]: Term 3 election: Requested vote from peers 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761), 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:50.166256 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a9ebd0c390436b8ace08089f719358" candidate_uuid: "eb0a49d2a951470e85adbcd797128e69" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: true dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3"
I20240502 07:59:50.166371 1066 raft_consensus.cc:3037] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Advancing to term 3
I20240502 07:59:50.167244 1066 raft_consensus.cc:2454] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb0a49d2a951470e85adbcd797128e69 in term 3.
I20240502 07:59:50.167516 1569 leader_election.cc:302] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e648af9dbb46fc881340efe1d1e3c3, eb0a49d2a951470e85adbcd797128e69; no voters:
I20240502 07:59:50.167703 1218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a9ebd0c390436b8ace08089f719358" candidate_uuid: "eb0a49d2a951470e85adbcd797128e69" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: true dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
I20240502 07:59:50.167846 1218 raft_consensus.cc:3032] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Stepping down as leader of term 2
I20240502 07:59:50.167901 1820 raft_consensus.cc:2781] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 07:59:50.167932 1218 raft_consensus.cc:729] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 27ec1b26c2f64cc682ed7ae12e133a9f, State: Running, Role: LEADER
I20240502 07:59:50.168118 1820 raft_consensus.cc:686] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [term 3 LEADER]: Becoming Leader. State: Replica: eb0a49d2a951470e85adbcd797128e69, State: Running, Role: LEADER
I20240502 07:59:50.168247 1218 consensus_queue.cc:260] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } }
I20240502 07:59:50.168292 1820 consensus_queue.cc:237] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } }
I20240502 07:59:50.168464 1218 raft_consensus.cc:3037] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Advancing to term 3
I20240502 07:59:50.169354 1218 raft_consensus.cc:2454] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb0a49d2a951470e85adbcd797128e69 in term 3.
I20240502 07:59:50.170022 961 catalog_manager.cc:5484] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 reported cstate change: term changed from 2 to 3, leader changed from 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131) to eb0a49d2a951470e85adbcd797128e69 (127.29.106.134). New cstate: current_term: 3 leader_uuid: "eb0a49d2a951470e85adbcd797128e69" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:50.237340 1734 raft_consensus.cc:979] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70: : Instructing follower 9d994926b3d54e6395a0d60df8a1db07 to start an election
I20240502 07:59:50.237488 1772 raft_consensus.cc:1067] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Signalling peer 9d994926b3d54e6395a0d60df8a1db07 to start an election
I20240502 07:59:50.237905 1454 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2c1b6523d6724a02acc9ae19f3f0255e"
dest_uuid: "9d994926b3d54e6395a0d60df8a1db07"
from {username='slave'} at 127.0.0.1:41168
I20240502 07:59:50.238026 1454 raft_consensus.cc:482] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:50.238096 1454 raft_consensus.cc:3037] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:50.238905 1454 raft_consensus.cc:504] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } }
I20240502 07:59:50.239293 1454 leader_election.cc:288] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [CANDIDATE]: Term 2 election: Requested vote from peers b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203), 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:50.239583 1139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1b6523d6724a02acc9ae19f3f0255e" candidate_uuid: "9d994926b3d54e6395a0d60df8a1db07" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
I20240502 07:59:50.239703 1139 raft_consensus.cc:3032] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:50.239744 1139 raft_consensus.cc:729] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b1e9945160714b16b2db91f90c5ccc70, State: Running, Role: LEADER
I20240502 07:59:50.239893 1139 consensus_queue.cc:260] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } }
I20240502 07:59:50.240072 1139 raft_consensus.cc:3037] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:50.241245 1139 raft_consensus.cc:2454] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d994926b3d54e6395a0d60df8a1db07 in term 2.
I20240502 07:59:50.241634 1417 leader_election.cc:302] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d994926b3d54e6395a0d60df8a1db07, b1e9945160714b16b2db91f90c5ccc70; no voters:
I20240502 07:59:50.242004 1824 raft_consensus.cc:2781] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:50.242184 1824 raft_consensus.cc:686] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [term 2 LEADER]: Becoming Leader. State: Replica: 9d994926b3d54e6395a0d60df8a1db07, State: Running, Role: LEADER
I20240502 07:59:50.242390 1824 consensus_queue.cc:237] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } }
I20240502 07:59:50.243707 1218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1b6523d6724a02acc9ae19f3f0255e" candidate_uuid: "9d994926b3d54e6395a0d60df8a1db07" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
I20240502 07:59:50.243820 1218 raft_consensus.cc:3037] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:50.243958 961 catalog_manager.cc:5484] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 reported cstate change: term changed from 1 to 2, leader changed from b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130) to 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132). New cstate: current_term: 2 leader_uuid: "9d994926b3d54e6395a0d60df8a1db07" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:50.244856 1218 raft_consensus.cc:2454] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d994926b3d54e6395a0d60df8a1db07 in term 2.
I20240502 07:59:50.247829 1808 raft_consensus.cc:979] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3: : Instructing follower 66df61031360493886f1170a809e079a to start an election
I20240502 07:59:50.247944 1808 raft_consensus.cc:1067] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Signalling peer 66df61031360493886f1170a809e079a to start an election
I20240502 07:59:50.248365 1530 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df4233b549b34b01b6f9662c7e7b3515"
dest_uuid: "66df61031360493886f1170a809e079a"
from {username='slave'} at 127.0.0.1:47864
I20240502 07:59:50.248499 1530 raft_consensus.cc:482] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:50.248592 1530 raft_consensus.cc:3037] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:50.249606 1530 raft_consensus.cc:504] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } }
I20240502 07:59:50.250079 1530 leader_election.cc:288] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [CANDIDATE]: Term 2 election: Requested vote from peers 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761), b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203)
I20240502 07:59:50.250577 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4233b549b34b01b6f9662c7e7b3515" candidate_uuid: "66df61031360493886f1170a809e079a" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3"
I20240502 07:59:50.250684 1066 raft_consensus.cc:3032] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:50.250730 1066 raft_consensus.cc:729] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77e648af9dbb46fc881340efe1d1e3c3, State: Running, Role: LEADER
I20240502 07:59:50.250867 1066 consensus_queue.cc:260] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } }
I20240502 07:59:50.251015 1066 raft_consensus.cc:3037] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:50.251904 1066 raft_consensus.cc:2454] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66df61031360493886f1170a809e079a in term 2.
I20240502 07:59:50.252197 1494 leader_election.cc:302] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66df61031360493886f1170a809e079a, 77e648af9dbb46fc881340efe1d1e3c3; no voters:
I20240502 07:59:50.252568 1828 raft_consensus.cc:2781] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:50.252791 1828 raft_consensus.cc:686] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [term 2 LEADER]: Becoming Leader. State: Replica: 66df61031360493886f1170a809e079a, State: Running, Role: LEADER
I20240502 07:59:50.252969 1828 consensus_queue.cc:237] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } }
I20240502 07:59:50.254559 961 catalog_manager.cc:5484] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a reported cstate change: term changed from 1 to 2, leader changed from 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129) to 66df61031360493886f1170a809e079a (127.29.106.133). New cstate: current_term: 2 leader_uuid: "66df61031360493886f1170a809e079a" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:50.256103 1143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4233b549b34b01b6f9662c7e7b3515" candidate_uuid: "66df61031360493886f1170a809e079a" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
I20240502 07:59:50.256211 1143 raft_consensus.cc:3037] T df4233b549b34b01b6f9662c7e7b3515 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:50.257105 1143 raft_consensus.cc:2454] T df4233b549b34b01b6f9662c7e7b3515 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66df61031360493886f1170a809e079a in term 2.
I20240502 07:59:50.297854 1781 raft_consensus.cc:979] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f: : Instructing follower b1e9945160714b16b2db91f90c5ccc70 to start an election
I20240502 07:59:50.297971 1781 raft_consensus.cc:1067] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Signalling peer b1e9945160714b16b2db91f90c5ccc70 to start an election
I20240502 07:59:50.298345 1143 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2ecebe8dad3946bd956afacf695fabaa"
dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
from {username='slave'} at 127.0.0.1:47886
I20240502 07:59:50.298516 1143 raft_consensus.cc:482] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:50.298583 1143 raft_consensus.cc:3037] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Advancing to term 3
I20240502 07:59:50.299506 1143 raft_consensus.cc:504] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } }
I20240502 07:59:50.299857 1143 leader_election.cc:288] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [CANDIDATE]: Term 3 election: Requested vote from peers 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761), 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:50.300160 1218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecebe8dad3946bd956afacf695fabaa" candidate_uuid: "b1e9945160714b16b2db91f90c5ccc70" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
I20240502 07:59:50.300294 1063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecebe8dad3946bd956afacf695fabaa" candidate_uuid: "b1e9945160714b16b2db91f90c5ccc70" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3"
I20240502 07:59:50.300295 1218 raft_consensus.cc:3032] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Stepping down as leader of term 2
I20240502 07:59:50.300426 1063 raft_consensus.cc:3037] T 2ecebe8dad3946bd956afacf695fabaa P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Advancing to term 3
I20240502 07:59:50.300452 1218 raft_consensus.cc:729] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 27ec1b26c2f64cc682ed7ae12e133a9f, State: Running, Role: LEADER
I20240502 07:59:50.300611 1218 consensus_queue.cc:260] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } }
I20240502 07:59:50.300812 1218 raft_consensus.cc:3037] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Advancing to term 3
I20240502 07:59:50.301461 1063 raft_consensus.cc:2454] T 2ecebe8dad3946bd956afacf695fabaa P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1e9945160714b16b2db91f90c5ccc70 in term 3.
I20240502 07:59:50.301692 1218 raft_consensus.cc:2454] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1e9945160714b16b2db91f90c5ccc70 in term 3.
I20240502 07:59:50.301728 1107 leader_election.cc:302] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e648af9dbb46fc881340efe1d1e3c3, b1e9945160714b16b2db91f90c5ccc70; no voters:
I20240502 07:59:50.301903 1772 raft_consensus.cc:2781] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 07:59:50.302006 1772 raft_consensus.cc:686] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 3 LEADER]: Becoming Leader. State: Replica: b1e9945160714b16b2db91f90c5ccc70, State: Running, Role: LEADER
I20240502 07:59:50.302139 1772 consensus_queue.cc:237] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } }
I20240502 07:59:50.304018 961 catalog_manager.cc:5484] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 reported cstate change: term changed from 2 to 3, leader changed from 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131) to b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130). New cstate: current_term: 3 leader_uuid: "b1e9945160714b16b2db91f90c5ccc70" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:50.450011 1143 raft_consensus.cc:1261] T d0d982dd52d246348fb3984fd1c3a40f P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 07:59:50.450536 1781 consensus_queue.cc:1035] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 07:59:50.453482 1063 raft_consensus.cc:1261] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 07:59:50.453928 1746 consensus_queue.cc:1035] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 07:59:50.534653 1605 raft_consensus.cc:1261] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [term 2 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240502 07:59:50.535082 1746 consensus_queue.cc:1035] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 07:59:50.537472 1139 raft_consensus.cc:1261] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240502 07:59:50.537792 1746 consensus_queue.cc:1035] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 07:59:50.575915 1063 raft_consensus.cc:1261] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 FOLLOWER]: Refusing update from remote peer eb0a49d2a951470e85adbcd797128e69: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240502 07:59:50.576352 1820 consensus_queue.cc:1035] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 07:59:50.579422 1211 raft_consensus.cc:1261] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Refusing update from remote peer eb0a49d2a951470e85adbcd797128e69: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240502 07:59:50.579891 1820 consensus_queue.cc:1035] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 07:59:50.674435 1218 raft_consensus.cc:1261] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Refusing update from remote peer 9d994926b3d54e6395a0d60df8a1db07: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240502 07:59:50.674906 1824 consensus_queue.cc:1035] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 07:59:50.677915 1143 raft_consensus.cc:1261] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Refusing update from remote peer 9d994926b3d54e6395a0d60df8a1db07: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240502 07:59:50.678453 1824 consensus_queue.cc:1035] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 07:59:50.690086 1063 raft_consensus.cc:1261] T 2ecebe8dad3946bd956afacf695fabaa P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 07:59:50.690544 1831 consensus_queue.cc:1035] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 07:59:50.693295 1218 raft_consensus.cc:1261] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 07:59:50.693800 1849 consensus_queue.cc:1035] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
W20240502 07:59:50.698772 1109 consensus_peers.cc:485] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 -> Peer 66df61031360493886f1170a809e079a (127.29.106.133:38717): Couldn't send request to peer 66df61031360493886f1170a809e079a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:50.757998 1063 raft_consensus.cc:1261] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Refusing update from remote peer 66df61031360493886f1170a809e079a: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240502 07:59:50.758538 1853 consensus_queue.cc:1035] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 07:59:50.761533 1139 raft_consensus.cc:1261] T df4233b549b34b01b6f9662c7e7b3515 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Refusing update from remote peer 66df61031360493886f1170a809e079a: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240502 07:59:50.761946 1857 consensus_queue.cc:1035] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 07:59:51.866570 1782 tablet_copy_client.cc:523] T 2ecebe8dad3946bd956afacf695fabaa P 66df61031360493886f1170a809e079a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:51.868108 1782 tablet_bootstrap.cc:492] T 2ecebe8dad3946bd956afacf695fabaa P 66df61031360493886f1170a809e079a: Bootstrap starting.
I20240502 07:59:51.889218 1782 tablet_bootstrap.cc:492] T 2ecebe8dad3946bd956afacf695fabaa P 66df61031360493886f1170a809e079a: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=92 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:51.889555 1782 tablet_bootstrap.cc:492] T 2ecebe8dad3946bd956afacf695fabaa P 66df61031360493886f1170a809e079a: Bootstrap complete.
I20240502 07:59:51.889746 1782 ts_tablet_manager.cc:1367] T 2ecebe8dad3946bd956afacf695fabaa P 66df61031360493886f1170a809e079a: Time spent bootstrapping tablet: real 0.022s user 0.021s sys 0.000s
I20240502 07:59:51.889907 1789 tablet_copy_client.cc:523] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:51.890321 1782 raft_consensus.cc:348] T 2ecebe8dad3946bd956afacf695fabaa P 66df61031360493886f1170a809e079a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } }
I20240502 07:59:51.890442 1782 raft_consensus.cc:729] T 2ecebe8dad3946bd956afacf695fabaa P 66df61031360493886f1170a809e079a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 66df61031360493886f1170a809e079a, State: Initialized, Role: LEARNER
I20240502 07:59:51.890574 1782 consensus_queue.cc:260] T 2ecebe8dad3946bd956afacf695fabaa P 66df61031360493886f1170a809e079a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } }
I20240502 07:59:51.891305 1782 ts_tablet_manager.cc:1398] T 2ecebe8dad3946bd956afacf695fabaa P 66df61031360493886f1170a809e079a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:51.891423 1789 tablet_bootstrap.cc:492] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07: Bootstrap starting.
I20240502 07:59:51.892436 1228 tablet_copy_service.cc:342] P 27ec1b26c2f64cc682ed7ae12e133a9f: Request end of tablet copy session 66df61031360493886f1170a809e079a-2ecebe8dad3946bd956afacf695fabaa received from {username='slave'} at 127.0.0.1:40304
I20240502 07:59:51.892518 1228 tablet_copy_service.cc:433] P 27ec1b26c2f64cc682ed7ae12e133a9f: ending tablet copy session 66df61031360493886f1170a809e079a-2ecebe8dad3946bd956afacf695fabaa on tablet 2ecebe8dad3946bd956afacf695fabaa with peer 66df61031360493886f1170a809e079a
I20240502 07:59:51.914284 1789 tablet_bootstrap.cc:492] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=109 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:51.914633 1789 tablet_bootstrap.cc:492] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07: Bootstrap complete.
I20240502 07:59:51.914845 1789 ts_tablet_manager.cc:1367] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07: Time spent bootstrapping tablet: real 0.023s user 0.017s sys 0.004s
I20240502 07:59:51.915365 1789 raft_consensus.cc:348] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } }
I20240502 07:59:51.915498 1789 raft_consensus.cc:729] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9d994926b3d54e6395a0d60df8a1db07, State: Initialized, Role: LEARNER
I20240502 07:59:51.915617 1789 consensus_queue.cc:260] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } }
I20240502 07:59:51.915925 1789 ts_tablet_manager.cc:1398] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 07:59:51.916447 1153 tablet_copy_service.cc:342] P b1e9945160714b16b2db91f90c5ccc70: Request end of tablet copy session 9d994926b3d54e6395a0d60df8a1db07-cdf0ff2967c040a589f915172d3f446b received from {username='slave'} at 127.0.0.1:47922
I20240502 07:59:51.916530 1153 tablet_copy_service.cc:433] P b1e9945160714b16b2db91f90c5ccc70: ending tablet copy session 9d994926b3d54e6395a0d60df8a1db07-cdf0ff2967c040a589f915172d3f446b on tablet cdf0ff2967c040a589f915172d3f446b with peer 9d994926b3d54e6395a0d60df8a1db07
I20240502 07:59:51.936391 1018 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:59:51.938217 1143 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5437f94679a841e3980053e69d624d76"
dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
mode: GRACEFUL
new_leader_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
from {username='slave'} at 127.0.0.1:47914
I20240502 07:59:51.938376 1143 raft_consensus.cc:595] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Received request to transfer leadership to TS 27ec1b26c2f64cc682ed7ae12e133a9f
I20240502 07:59:51.939218 1218 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "58c5f52a420c467684b2cb50972e7c99"
dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
mode: GRACEFUL
new_leader_uuid: "77e648af9dbb46fc881340efe1d1e3c3"
from {username='slave'} at 127.0.0.1:40314
I20240502 07:59:51.939360 1218 raft_consensus.cc:595] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 LEADER]: Received request to transfer leadership to TS 77e648af9dbb46fc881340efe1d1e3c3
I20240502 07:59:51.940119 1218 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0d982dd52d246348fb3984fd1c3a40f"
dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
mode: GRACEFUL
new_leader_uuid: "77e648af9dbb46fc881340efe1d1e3c3"
from {username='slave'} at 127.0.0.1:40314
I20240502 07:59:51.940209 1218 raft_consensus.cc:595] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Received request to transfer leadership to TS 77e648af9dbb46fc881340efe1d1e3c3
I20240502 07:59:51.940486 1018 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240502 07:59:51.940630 1018 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:59:51.942027 1795 tablet_copy_client.cc:523] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:51.943372 1795 tablet_bootstrap.cc:492] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a: Bootstrap starting.
I20240502 07:59:51.966225 1795 tablet_bootstrap.cc:492] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=114 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:51.966506 1795 tablet_bootstrap.cc:492] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a: Bootstrap complete.
I20240502 07:59:51.966694 1795 ts_tablet_manager.cc:1367] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a: Time spent bootstrapping tablet: real 0.023s user 0.020s sys 0.002s
I20240502 07:59:51.967206 1795 raft_consensus.cc:348] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } }
I20240502 07:59:51.967327 1795 raft_consensus.cc:729] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66df61031360493886f1170a809e079a, State: Initialized, Role: LEARNER
I20240502 07:59:51.967466 1795 consensus_queue.cc:260] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } }
I20240502 07:59:51.967782 1795 ts_tablet_manager.cc:1398] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 07:59:51.968240 1228 tablet_copy_service.cc:342] P 27ec1b26c2f64cc682ed7ae12e133a9f: Request end of tablet copy session 66df61031360493886f1170a809e079a-58c5f52a420c467684b2cb50972e7c99 received from {username='slave'} at 127.0.0.1:40304
I20240502 07:59:51.968360 1228 tablet_copy_service.cc:433] P 27ec1b26c2f64cc682ed7ae12e133a9f: ending tablet copy session 66df61031360493886f1170a809e079a-58c5f52a420c467684b2cb50972e7c99 on tablet 58c5f52a420c467684b2cb50972e7c99 with peer 66df61031360493886f1170a809e079a
I20240502 07:59:51.982950 1801 tablet_copy_client.cc:523] T fda98cfe14f947969fb41e26c75bc384 P eb0a49d2a951470e85adbcd797128e69: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:51.984747 1801 tablet_bootstrap.cc:492] T fda98cfe14f947969fb41e26c75bc384 P eb0a49d2a951470e85adbcd797128e69: Bootstrap starting.
I20240502 07:59:51.998986 1809 tablet_copy_client.cc:523] T 5437f94679a841e3980053e69d624d76 P eb0a49d2a951470e85adbcd797128e69: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:52.000803 1809 tablet_bootstrap.cc:492] T 5437f94679a841e3980053e69d624d76 P eb0a49d2a951470e85adbcd797128e69: Bootstrap starting.
I20240502 07:59:52.008773 1801 tablet_bootstrap.cc:492] T fda98cfe14f947969fb41e26c75bc384 P eb0a49d2a951470e85adbcd797128e69: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=110 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:52.009116 1801 tablet_bootstrap.cc:492] T fda98cfe14f947969fb41e26c75bc384 P eb0a49d2a951470e85adbcd797128e69: Bootstrap complete.
I20240502 07:59:52.009110 1815 tablet_copy_client.cc:523] T c7c1536618ae4e74aa79349d04201fb9 P 9d994926b3d54e6395a0d60df8a1db07: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:52.010807 1815 tablet_bootstrap.cc:492] T c7c1536618ae4e74aa79349d04201fb9 P 9d994926b3d54e6395a0d60df8a1db07: Bootstrap starting.
I20240502 07:59:52.015626 1801 ts_tablet_manager.cc:1367] T fda98cfe14f947969fb41e26c75bc384 P eb0a49d2a951470e85adbcd797128e69: Time spent bootstrapping tablet: real 0.031s user 0.024s sys 0.000s
I20240502 07:59:52.024818 1454 raft_consensus.cc:1203] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20240502 07:59:52.025652 1809 tablet_bootstrap.cc:492] T 5437f94679a841e3980053e69d624d76 P eb0a49d2a951470e85adbcd797128e69: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=112 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:52.025794 1801 raft_consensus.cc:348] T fda98cfe14f947969fb41e26c75bc384 P eb0a49d2a951470e85adbcd797128e69 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } }
I20240502 07:59:52.025952 1801 raft_consensus.cc:729] T fda98cfe14f947969fb41e26c75bc384 P eb0a49d2a951470e85adbcd797128e69 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb0a49d2a951470e85adbcd797128e69, State: Initialized, Role: LEARNER
I20240502 07:59:52.026011 1809 tablet_bootstrap.cc:492] T 5437f94679a841e3980053e69d624d76 P eb0a49d2a951470e85adbcd797128e69: Bootstrap complete.
I20240502 07:59:52.026083 1801 consensus_queue.cc:260] T fda98cfe14f947969fb41e26c75bc384 P eb0a49d2a951470e85adbcd797128e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 1.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } }
I20240502 07:59:52.026345 1809 ts_tablet_manager.cc:1367] T 5437f94679a841e3980053e69d624d76 P eb0a49d2a951470e85adbcd797128e69: Time spent bootstrapping tablet: real 0.026s user 0.021s sys 0.004s
I20240502 07:59:52.026876 1801 ts_tablet_manager.cc:1398] T fda98cfe14f947969fb41e26c75bc384 P eb0a49d2a951470e85adbcd797128e69: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240502 07:59:52.026831 1809 raft_consensus.cc:348] T 5437f94679a841e3980053e69d624d76 P eb0a49d2a951470e85adbcd797128e69 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } }
I20240502 07:59:52.027069 1809 raft_consensus.cc:729] T 5437f94679a841e3980053e69d624d76 P eb0a49d2a951470e85adbcd797128e69 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb0a49d2a951470e85adbcd797128e69, State: Initialized, Role: LEARNER
I20240502 07:59:52.027218 1809 consensus_queue.cc:260] T 5437f94679a841e3980053e69d624d76 P eb0a49d2a951470e85adbcd797128e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } }
I20240502 07:59:52.027307 1077 tablet_copy_service.cc:342] P 77e648af9dbb46fc881340efe1d1e3c3: Request end of tablet copy session eb0a49d2a951470e85adbcd797128e69-fda98cfe14f947969fb41e26c75bc384 received from {username='slave'} at 127.0.0.1:54320
I20240502 07:59:52.027375 1077 tablet_copy_service.cc:433] P 77e648af9dbb46fc881340efe1d1e3c3: ending tablet copy session eb0a49d2a951470e85adbcd797128e69-fda98cfe14f947969fb41e26c75bc384 on tablet fda98cfe14f947969fb41e26c75bc384 with peer eb0a49d2a951470e85adbcd797128e69
I20240502 07:59:52.027583 1809 ts_tablet_manager.cc:1398] T 5437f94679a841e3980053e69d624d76 P eb0a49d2a951470e85adbcd797128e69: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:52.028342 1153 tablet_copy_service.cc:342] P b1e9945160714b16b2db91f90c5ccc70: Request end of tablet copy session eb0a49d2a951470e85adbcd797128e69-5437f94679a841e3980053e69d624d76 received from {username='slave'} at 127.0.0.1:47924
I20240502 07:59:52.028475 1153 tablet_copy_service.cc:433] P b1e9945160714b16b2db91f90c5ccc70: ending tablet copy session eb0a49d2a951470e85adbcd797128e69-5437f94679a841e3980053e69d624d76 on tablet 5437f94679a841e3980053e69d624d76 with peer eb0a49d2a951470e85adbcd797128e69
I20240502 07:59:52.036610 1530 raft_consensus.cc:1203] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29] Dedup: 1.29->[]
I20240502 07:59:52.040042 1815 tablet_bootstrap.cc:492] T c7c1536618ae4e74aa79349d04201fb9 P 9d994926b3d54e6395a0d60df8a1db07: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=122 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:52.040380 1815 tablet_bootstrap.cc:492] T c7c1536618ae4e74aa79349d04201fb9 P 9d994926b3d54e6395a0d60df8a1db07: Bootstrap complete.
I20240502 07:59:52.040591 1815 ts_tablet_manager.cc:1367] T c7c1536618ae4e74aa79349d04201fb9 P 9d994926b3d54e6395a0d60df8a1db07: Time spent bootstrapping tablet: real 0.030s user 0.019s sys 0.008s
I20240502 07:59:52.041113 1815 raft_consensus.cc:348] T c7c1536618ae4e74aa79349d04201fb9 P 9d994926b3d54e6395a0d60df8a1db07 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } }
I20240502 07:59:52.041224 1815 raft_consensus.cc:729] T c7c1536618ae4e74aa79349d04201fb9 P 9d994926b3d54e6395a0d60df8a1db07 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9d994926b3d54e6395a0d60df8a1db07, State: Initialized, Role: LEARNER
I20240502 07:59:52.041338 1815 consensus_queue.cc:260] T c7c1536618ae4e74aa79349d04201fb9 P 9d994926b3d54e6395a0d60df8a1db07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 1.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } }
I20240502 07:59:52.041656 1815 ts_tablet_manager.cc:1398] T c7c1536618ae4e74aa79349d04201fb9 P 9d994926b3d54e6395a0d60df8a1db07: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:52.042099 1077 tablet_copy_service.cc:342] P 77e648af9dbb46fc881340efe1d1e3c3: Request end of tablet copy session 9d994926b3d54e6395a0d60df8a1db07-c7c1536618ae4e74aa79349d04201fb9 received from {username='slave'} at 127.0.0.1:48350
I20240502 07:59:52.042172 1077 tablet_copy_service.cc:433] P 77e648af9dbb46fc881340efe1d1e3c3: ending tablet copy session 9d994926b3d54e6395a0d60df8a1db07-c7c1536618ae4e74aa79349d04201fb9 on tablet c7c1536618ae4e74aa79349d04201fb9 with peer 9d994926b3d54e6395a0d60df8a1db07
I20240502 07:59:52.085708 1839 raft_consensus.cc:979] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f: : Instructing follower 77e648af9dbb46fc881340efe1d1e3c3 to start an election
I20240502 07:59:52.085850 1839 raft_consensus.cc:1067] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Signalling peer 77e648af9dbb46fc881340efe1d1e3c3 to start an election
I20240502 07:59:52.086213 1066 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d0d982dd52d246348fb3984fd1c3a40f"
dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3"
from {username='slave'} at 127.0.0.1:48316
I20240502 07:59:52.086350 1066 raft_consensus.cc:482] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:52.086423 1066 raft_consensus.cc:3037] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Advancing to term 3
I20240502 07:59:52.087469 1066 raft_consensus.cc:504] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:52.087806 1066 leader_election.cc:288] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [CANDIDATE]: Term 3 election: Requested vote from peers b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203), 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:52.088092 1218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d982dd52d246348fb3984fd1c3a40f" candidate_uuid: "77e648af9dbb46fc881340efe1d1e3c3" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
I20240502 07:59:52.088197 1218 raft_consensus.cc:3032] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Stepping down as leader of term 2
I20240502 07:59:52.088174 1143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d982dd52d246348fb3984fd1c3a40f" candidate_uuid: "77e648af9dbb46fc881340efe1d1e3c3" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
I20240502 07:59:52.088246 1218 raft_consensus.cc:729] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 27ec1b26c2f64cc682ed7ae12e133a9f, State: Running, Role: LEADER
I20240502 07:59:52.088303 1143 raft_consensus.cc:3037] T d0d982dd52d246348fb3984fd1c3a40f P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Advancing to term 3
I20240502 07:59:52.088410 1218 consensus_queue.cc:260] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:52.088569 1218 raft_consensus.cc:3037] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Advancing to term 3
I20240502 07:59:52.089133 1143 raft_consensus.cc:2454] T d0d982dd52d246348fb3984fd1c3a40f P b1e9945160714b16b2db91f90c5ccc70 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e648af9dbb46fc881340efe1d1e3c3 in term 3.
I20240502 07:59:52.089255 1218 raft_consensus.cc:2454] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e648af9dbb46fc881340efe1d1e3c3 in term 3.
I20240502 07:59:52.089390 1030 leader_election.cc:302] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e648af9dbb46fc881340efe1d1e3c3, b1e9945160714b16b2db91f90c5ccc70; no voters:
I20240502 07:59:52.089555 1725 raft_consensus.cc:2781] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 07:59:52.089779 1725 raft_consensus.cc:686] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 LEADER]: Becoming Leader. State: Replica: 77e648af9dbb46fc881340efe1d1e3c3, State: Running, Role: LEADER
I20240502 07:59:52.089915 1725 consensus_queue.cc:237] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:52.091324 955 catalog_manager.cc:5484] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 reported cstate change: term changed from 2 to 3, leader changed from 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131) to 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129). New cstate: current_term: 3 leader_uuid: "77e648af9dbb46fc881340efe1d1e3c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:52.129340 1454 raft_consensus.cc:1203] T c7c1536618ae4e74aa79349d04201fb9 P 9d994926b3d54e6395a0d60df8a1db07 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.31->[1.32-1.32] Dedup: 1.32->[]
I20240502 07:59:52.194068 1530 raft_consensus.cc:3037] T 2ecebe8dad3946bd956afacf695fabaa P 66df61031360493886f1170a809e079a [term 2 LEARNER]: Advancing to term 3
I20240502 07:59:52.204015 1875 raft_consensus.cc:979] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70: : Instructing follower 27ec1b26c2f64cc682ed7ae12e133a9f to start an election
I20240502 07:59:52.204150 1875 raft_consensus.cc:1067] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Signalling peer 27ec1b26c2f64cc682ed7ae12e133a9f to start an election
I20240502 07:59:52.204718 1218 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5437f94679a841e3980053e69d624d76"
dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
from {username='slave'} at 127.0.0.1:52044
I20240502 07:59:52.204861 1218 raft_consensus.cc:482] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:52.204928 1218 raft_consensus.cc:3037] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:52.205987 1218 raft_consensus.cc:504] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } }
I20240502 07:59:52.206359 1218 leader_election.cc:288] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [CANDIDATE]: Term 2 election: Requested vote from peers 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761), b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203)
I20240502 07:59:52.206630 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5437f94679a841e3980053e69d624d76" candidate_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3"
I20240502 07:59:52.206732 1066 raft_consensus.cc:3037] T 5437f94679a841e3980053e69d624d76 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:52.206845 1139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5437f94679a841e3980053e69d624d76" candidate_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
I20240502 07:59:52.206933 1139 raft_consensus.cc:3032] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:52.206977 1139 raft_consensus.cc:729] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b1e9945160714b16b2db91f90c5ccc70, State: Running, Role: LEADER
I20240502 07:59:52.207111 1139 consensus_queue.cc:260] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } }
I20240502 07:59:52.207276 1139 raft_consensus.cc:3037] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:52.207748 1066 raft_consensus.cc:2454] T 5437f94679a841e3980053e69d624d76 P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27ec1b26c2f64cc682ed7ae12e133a9f in term 2.
I20240502 07:59:52.208060 1182 leader_election.cc:302] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ec1b26c2f64cc682ed7ae12e133a9f, 77e648af9dbb46fc881340efe1d1e3c3; no voters:
I20240502 07:59:52.208176 1139 raft_consensus.cc:2454] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27ec1b26c2f64cc682ed7ae12e133a9f in term 2.
I20240502 07:59:52.208225 1866 raft_consensus.cc:2781] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:52.208323 1866 raft_consensus.cc:686] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Becoming Leader. State: Replica: 27ec1b26c2f64cc682ed7ae12e133a9f, State: Running, Role: LEADER
I20240502 07:59:52.208456 1866 consensus_queue.cc:237] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } }
I20240502 07:59:52.210120 955 catalog_manager.cc:5484] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f reported cstate change: term changed from 1 to 2, leader changed from b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130) to 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131). New cstate: current_term: 2 leader_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:52.215459 1218 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5437f94679a841e3980053e69d624d76"
dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52036
I20240502 07:59:52.215605 1218 raft_consensus.cc:595] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Received request to transfer leadership
I20240502 07:59:52.221053 1218 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5437f94679a841e3980053e69d624d76"
dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52036
I20240502 07:59:52.221192 1218 raft_consensus.cc:595] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Received request to transfer leadership
W20240502 07:59:52.221493 951 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 5437f94679a841e3980053e69d624d76 already in progress [suppressed 2 similar messages]
I20240502 07:59:52.222270 1218 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5437f94679a841e3980053e69d624d76"
dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52036
I20240502 07:59:52.222383 1218 raft_consensus.cc:595] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Received request to transfer leadership
W20240502 07:59:52.222762 1013 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 5437f94679a841e3980053e69d624d76 already in progress
W20240502 07:59:52.222822 1013 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 5437f94679a841e3980053e69d624d76 already in progress
I20240502 07:59:52.243969 1839 raft_consensus.cc:979] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f: : Instructing follower 77e648af9dbb46fc881340efe1d1e3c3 to start an election
I20240502 07:59:52.244096 1839 raft_consensus.cc:1067] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 LEADER]: Signalling peer 77e648af9dbb46fc881340efe1d1e3c3 to start an election
I20240502 07:59:52.244472 1066 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "58c5f52a420c467684b2cb50972e7c99"
dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3"
from {username='slave'} at 127.0.0.1:48316
I20240502 07:59:52.244661 1066 raft_consensus.cc:482] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:52.244729 1066 raft_consensus.cc:3037] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:52.245792 1066 raft_consensus.cc:504] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } }
I20240502 07:59:52.246166 1066 leader_election.cc:288] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [CANDIDATE]: Term 2 election: Requested vote from peers b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203), 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:52.246390 1139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c5f52a420c467684b2cb50972e7c99" candidate_uuid: "77e648af9dbb46fc881340efe1d1e3c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
I20240502 07:59:52.246495 1139 raft_consensus.cc:3037] T 58c5f52a420c467684b2cb50972e7c99 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:52.246520 1218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c5f52a420c467684b2cb50972e7c99" candidate_uuid: "77e648af9dbb46fc881340efe1d1e3c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
I20240502 07:59:52.246646 1218 raft_consensus.cc:3032] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:52.246701 1218 raft_consensus.cc:729] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 27ec1b26c2f64cc682ed7ae12e133a9f, State: Running, Role: LEADER
I20240502 07:59:52.246834 1218 consensus_queue.cc:260] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } }
I20240502 07:59:52.246994 1218 raft_consensus.cc:3037] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:52.247577 1139 raft_consensus.cc:2454] T 58c5f52a420c467684b2cb50972e7c99 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e648af9dbb46fc881340efe1d1e3c3 in term 2.
I20240502 07:59:52.247807 1218 raft_consensus.cc:2454] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e648af9dbb46fc881340efe1d1e3c3 in term 2.
I20240502 07:59:52.248054 1033 leader_election.cc:302] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ec1b26c2f64cc682ed7ae12e133a9f, 77e648af9dbb46fc881340efe1d1e3c3; no voters:
I20240502 07:59:52.248353 1725 raft_consensus.cc:2781] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:52.248446 1725 raft_consensus.cc:686] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 LEADER]: Becoming Leader. State: Replica: 77e648af9dbb46fc881340efe1d1e3c3, State: Running, Role: LEADER
I20240502 07:59:52.248569 1725 consensus_queue.cc:237] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } }
I20240502 07:59:52.250339 955 catalog_manager.cc:5484] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 reported cstate change: term changed from 1 to 2, leader changed from 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131) to 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129). New cstate: current_term: 2 leader_uuid: "77e648af9dbb46fc881340efe1d1e3c3" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:52.266767 1605 raft_consensus.cc:1203] T fda98cfe14f947969fb41e26c75bc384 P eb0a49d2a951470e85adbcd797128e69 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.31->[1.32-1.32] Dedup: 1.32->[]
I20240502 07:59:52.533901 1867 raft_consensus.cc:1050] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3: attempting to promote NON_VOTER 9d994926b3d54e6395a0d60df8a1db07 to VOTER
I20240502 07:59:52.534355 1867 consensus_queue.cc:237] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.32, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } }
I20240502 07:59:52.535631 1454 raft_consensus.cc:1261] T c7c1536618ae4e74aa79349d04201fb9 P 9d994926b3d54e6395a0d60df8a1db07 [term 1 LEARNER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20240502 07:59:52.535755 1211 raft_consensus.cc:1261] T c7c1536618ae4e74aa79349d04201fb9 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20240502 07:59:52.535889 1530 raft_consensus.cc:1261] T c7c1536618ae4e74aa79349d04201fb9 P 66df61031360493886f1170a809e079a [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20240502 07:59:52.536204 1881 consensus_queue.cc:1035] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 07:59:52.536540 1725 consensus_queue.cc:1035] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 07:59:52.536919 1879 consensus_queue.cc:1035] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 07:59:52.539171 1725 raft_consensus.cc:2932] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Committing config change with OpId 1.33: config changed from index 32 to 33, 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:52.539589 1454 raft_consensus.cc:2932] T c7c1536618ae4e74aa79349d04201fb9 P 9d994926b3d54e6395a0d60df8a1db07 [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index 32 to 33, 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:52.541240 1530 raft_consensus.cc:2932] T c7c1536618ae4e74aa79349d04201fb9 P 66df61031360493886f1170a809e079a [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index 32 to 33, 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:52.541289 1211 raft_consensus.cc:2932] T c7c1536618ae4e74aa79349d04201fb9 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index 32 to 33, 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:52.541731 1143 raft_consensus.cc:1261] T d0d982dd52d246348fb3984fd1c3a40f P b1e9945160714b16b2db91f90c5ccc70 [term 3 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 07:59:52.542079 1879 consensus_queue.cc:1035] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:52.543334 955 catalog_manager.cc:5484] T c7c1536618ae4e74aa79349d04201fb9 P 9d994926b3d54e6395a0d60df8a1db07 reported cstate change: config changed from index 32 to 33, 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "77e648af9dbb46fc881340efe1d1e3c3" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:52.547099 1063 consensus_queue.cc:237] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 1.33, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } }
I20240502 07:59:52.548063 1211 raft_consensus.cc:1261] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 07:59:52.548520 1454 raft_consensus.cc:1261] T c7c1536618ae4e74aa79349d04201fb9 P 9d994926b3d54e6395a0d60df8a1db07 [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 1 index: 34. (index mismatch)
I20240502 07:59:52.548544 1867 consensus_queue.cc:1035] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:52.548971 1867 consensus_queue.cc:1035] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 07:59:52.549108 1530 raft_consensus.cc:1261] T c7c1536618ae4e74aa79349d04201fb9 P 66df61031360493886f1170a809e079a [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 1 index: 34. (index mismatch)
I20240502 07:59:52.550400 1867 consensus_queue.cc:1035] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 07:59:52.550948 1772 raft_consensus.cc:1050] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70: attempting to promote NON_VOTER 9d994926b3d54e6395a0d60df8a1db07 to VOTER
I20240502 07:59:52.551316 1772 consensus_queue.cc:237] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } }
I20240502 07:59:52.551661 1867 raft_consensus.cc:2932] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Committing config change with OpId 1.34: config changed from index 33 to 34, VOTER 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:52.552130 1454 raft_consensus.cc:2932] T c7c1536618ae4e74aa79349d04201fb9 P 9d994926b3d54e6395a0d60df8a1db07 [term 1 FOLLOWER]: Committing config change with OpId 1.34: config changed from index 33 to 34, VOTER 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:52.553278 1211 raft_consensus.cc:1261] T cdf0ff2967c040a589f915172d3f446b P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 07:59:52.553607 1063 raft_consensus.cc:1261] T cdf0ff2967c040a589f915172d3f446b P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 07:59:52.553658 1772 consensus_queue.cc:1035] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 07:59:52.553608 1453 raft_consensus.cc:1261] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07 [term 1 LEARNER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 07:59:52.554561 1892 consensus_queue.cc:1035] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 07:59:52.554783 1530 raft_consensus.cc:2932] T c7c1536618ae4e74aa79349d04201fb9 P 66df61031360493886f1170a809e079a [term 1 FOLLOWER]: Committing config change with OpId 1.34: config changed from index 33 to 34, VOTER 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:52.555346 962 catalog_manager.cc:5484] T c7c1536618ae4e74aa79349d04201fb9 P 9d994926b3d54e6395a0d60df8a1db07 reported cstate change: config changed from index 33 to 34, VOTER 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131) evicted. New cstate: current_term: 1 leader_uuid: "77e648af9dbb46fc881340efe1d1e3c3" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:52.554989 1892 consensus_queue.cc:1035] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 07:59:52.556313 949 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c7c1536618ae4e74aa79349d04201fb9 with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:52.558452 1207 tablet_service.cc:1514] Processing DeleteTablet for tablet c7c1536618ae4e74aa79349d04201fb9 with delete_type TABLET_DATA_TOMBSTONED (TS 27ec1b26c2f64cc682ed7ae12e133a9f not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:52036
I20240502 07:59:52.558815 1895 tablet_replica.cc:330] T c7c1536618ae4e74aa79349d04201fb9 P 27ec1b26c2f64cc682ed7ae12e133a9f: stopping tablet replica
I20240502 07:59:52.558784 1772 raft_consensus.cc:2932] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:52.558969 1895 raft_consensus.cc:2229] T c7c1536618ae4e74aa79349d04201fb9 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:52.559178 1895 raft_consensus.cc:2258] T c7c1536618ae4e74aa79349d04201fb9 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:52.560298 1895 ts_tablet_manager.cc:1875] T c7c1536618ae4e74aa79349d04201fb9 P 27ec1b26c2f64cc682ed7ae12e133a9f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:52.560518 1453 raft_consensus.cc:2932] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:52.562227 1895 ts_tablet_manager.cc:1888] T c7c1536618ae4e74aa79349d04201fb9 P 27ec1b26c2f64cc682ed7ae12e133a9f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.33
I20240502 07:59:52.562302 1895 log.cc:1196] T c7c1536618ae4e74aa79349d04201fb9 P 27ec1b26c2f64cc682ed7ae12e133a9f: Deleting WAL directory at /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-2-root/wals/c7c1536618ae4e74aa79349d04201fb9
I20240502 07:59:52.562393 1211 raft_consensus.cc:2932] T cdf0ff2967c040a589f915172d3f446b P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:52.562716 960 catalog_manager.cc:5484] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07 reported cstate change: config changed from index 28 to 29, 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b1e9945160714b16b2db91f90c5ccc70" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:52.563807 951 catalog_manager.cc:4830] TS 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737): tablet c7c1536618ae4e74aa79349d04201fb9 (table test-workload [id=c4044eda63f741838cbc708a8054d3b9]) successfully deleted
I20240502 07:59:52.564604 1066 raft_consensus.cc:2932] T cdf0ff2967c040a589f915172d3f446b P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
W20240502 07:59:52.566635 1891 consensus_queue.cc:1159] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "77e648af9dbb46fc881340efe1d1e3c3" responder_term: 1 status { last_received { term: 1 index: 29 } last_committed_idx: 29 last_received_current_leader { term: 1 index: 29 } }
I20240502 07:59:52.566794 1143 consensus_queue.cc:237] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } }
I20240502 07:59:52.568015 1453 raft_consensus.cc:1261] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07 [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 07:59:52.568374 1891 consensus_queue.cc:1035] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:52.568382 1211 raft_consensus.cc:1261] T cdf0ff2967c040a589f915172d3f446b P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 07:59:52.568769 1891 consensus_queue.cc:1035] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:52.570006 1875 raft_consensus.cc:2932] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:52.570289 1453 raft_consensus.cc:2932] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:52.572073 1211 raft_consensus.cc:2932] T cdf0ff2967c040a589f915172d3f446b P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:52.572074 948 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cdf0ff2967c040a589f915172d3f446b with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:52.572774 962 catalog_manager.cc:5484] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 reported cstate change: config changed from index 29 to 30, VOTER 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129) evicted. New cstate: current_term: 1 leader_uuid: "b1e9945160714b16b2db91f90c5ccc70" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:52.575146 1056 tablet_service.cc:1514] Processing DeleteTablet for tablet cdf0ff2967c040a589f915172d3f446b with delete_type TABLET_DATA_TOMBSTONED (TS 77e648af9dbb46fc881340efe1d1e3c3 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:48298
I20240502 07:59:52.575620 1897 tablet_replica.cc:330] T cdf0ff2967c040a589f915172d3f446b P 77e648af9dbb46fc881340efe1d1e3c3: stopping tablet replica
I20240502 07:59:52.575755 1897 raft_consensus.cc:2229] T cdf0ff2967c040a589f915172d3f446b P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:52.575928 1897 raft_consensus.cc:2258] T cdf0ff2967c040a589f915172d3f446b P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:52.577122 1897 ts_tablet_manager.cc:1875] T cdf0ff2967c040a589f915172d3f446b P 77e648af9dbb46fc881340efe1d1e3c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:52.579270 1897 ts_tablet_manager.cc:1888] T cdf0ff2967c040a589f915172d3f446b P 77e648af9dbb46fc881340efe1d1e3c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20240502 07:59:52.579356 1897 log.cc:1196] T cdf0ff2967c040a589f915172d3f446b P 77e648af9dbb46fc881340efe1d1e3c3: Deleting WAL directory at /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-0-root/wals/cdf0ff2967c040a589f915172d3f446b
I20240502 07:59:52.580008 949 catalog_manager.cc:4830] TS 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761): tablet cdf0ff2967c040a589f915172d3f446b (table test-workload [id=c4044eda63f741838cbc708a8054d3b9]) successfully deleted
I20240502 07:59:52.708711 1139 raft_consensus.cc:1261] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 07:59:52.709095 1866 consensus_queue.cc:1035] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:52.710661 1839 raft_consensus.cc:979] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f: : Instructing follower b1e9945160714b16b2db91f90c5ccc70 to start an election
I20240502 07:59:52.710816 1866 raft_consensus.cc:1067] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Signalling peer b1e9945160714b16b2db91f90c5ccc70 to start an election
I20240502 07:59:52.711192 1143 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5437f94679a841e3980053e69d624d76"
dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
from {username='slave'} at 127.0.0.1:47886
I20240502 07:59:52.711313 1143 raft_consensus.cc:482] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:52.711391 1143 raft_consensus.cc:3037] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Advancing to term 3
I20240502 07:59:52.712527 1143 raft_consensus.cc:504] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } }
I20240502 07:59:52.712886 1143 leader_election.cc:288] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [CANDIDATE]: Term 3 election: Requested vote from peers 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761), 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:52.713217 1211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5437f94679a841e3980053e69d624d76" candidate_uuid: "b1e9945160714b16b2db91f90c5ccc70" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
I20240502 07:59:52.713254 1063 raft_consensus.cc:1261] T 5437f94679a841e3980053e69d624d76 P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 07:59:52.713330 1211 raft_consensus.cc:3032] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Stepping down as leader of term 2
I20240502 07:59:52.713369 1211 raft_consensus.cc:729] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 27ec1b26c2f64cc682ed7ae12e133a9f, State: Running, Role: LEADER
I20240502 07:59:52.713517 1211 consensus_queue.cc:260] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } }
I20240502 07:59:52.713737 1211 raft_consensus.cc:3037] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Advancing to term 3
I20240502 07:59:52.713200 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5437f94679a841e3980053e69d624d76" candidate_uuid: "b1e9945160714b16b2db91f90c5ccc70" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3"
I20240502 07:59:52.714062 1066 raft_consensus.cc:3037] T 5437f94679a841e3980053e69d624d76 P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Advancing to term 3
I20240502 07:59:52.714747 1211 raft_consensus.cc:2454] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1e9945160714b16b2db91f90c5ccc70 in term 3.
I20240502 07:59:52.714983 1066 raft_consensus.cc:2454] T 5437f94679a841e3980053e69d624d76 P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1e9945160714b16b2db91f90c5ccc70 in term 3.
I20240502 07:59:52.715034 1109 leader_election.cc:302] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ec1b26c2f64cc682ed7ae12e133a9f, b1e9945160714b16b2db91f90c5ccc70; no voters:
I20240502 07:59:52.715214 1892 raft_consensus.cc:2781] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 07:59:52.715296 1892 raft_consensus.cc:686] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [term 3 LEADER]: Becoming Leader. State: Replica: b1e9945160714b16b2db91f90c5ccc70, State: Running, Role: LEADER
I20240502 07:59:52.715423 1892 consensus_queue.cc:237] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } }
I20240502 07:59:52.717026 960 catalog_manager.cc:5484] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 reported cstate change: term changed from 2 to 3, leader changed from 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131) to b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130). New cstate: current_term: 3 leader_uuid: "b1e9945160714b16b2db91f90c5ccc70" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:52.728878 1875 raft_consensus.cc:1050] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70: attempting to promote NON_VOTER 66df61031360493886f1170a809e079a to VOTER
I20240502 07:59:52.729331 1875 consensus_queue.cc:237] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 28, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } }
I20240502 07:59:52.730649 1530 raft_consensus.cc:1261] T 2ecebe8dad3946bd956afacf695fabaa P 66df61031360493886f1170a809e079a [term 3 LEARNER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 07:59:52.730667 1211 raft_consensus.cc:1261] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 07:59:52.730906 1066 raft_consensus.cc:1261] T 2ecebe8dad3946bd956afacf695fabaa P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 07:59:52.731089 1892 consensus_queue.cc:1035] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:52.731287 1875 consensus_queue.cc:1035] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:52.731444 1772 consensus_queue.cc:1035] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:52.733358 1892 raft_consensus.cc:2932] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index 28 to 30, 66df61031360493886f1170a809e079a (127.29.106.133) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:52.733996 1211 raft_consensus.cc:2932] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 28 to 30, 66df61031360493886f1170a809e079a (127.29.106.133) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:52.734967 1063 raft_consensus.cc:2932] T 2ecebe8dad3946bd956afacf695fabaa P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 28 to 30, 66df61031360493886f1170a809e079a (127.29.106.133) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:52.735108 1530 raft_consensus.cc:2932] T 2ecebe8dad3946bd956afacf695fabaa P 66df61031360493886f1170a809e079a [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 28 to 30, 66df61031360493886f1170a809e079a (127.29.106.133) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:52.737118 960 catalog_manager.cc:5484] T 2ecebe8dad3946bd956afacf695fabaa P 77e648af9dbb46fc881340efe1d1e3c3 reported cstate change: config changed from index 28 to 30, 66df61031360493886f1170a809e079a (127.29.106.133) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "b1e9945160714b16b2db91f90c5ccc70" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:52.740064 1139 consensus_queue.cc:237] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } }
I20240502 07:59:52.741370 1530 raft_consensus.cc:1261] T 2ecebe8dad3946bd956afacf695fabaa P 66df61031360493886f1170a809e079a [term 3 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 07:59:52.741681 1891 consensus_queue.cc:1035] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 07:59:52.741879 1211 raft_consensus.cc:1261] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 07:59:52.742202 1891 consensus_queue.cc:1035] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 07:59:52.743149 1892 raft_consensus.cc:2932] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index 30 to 31, VOTER 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:52.743568 1530 raft_consensus.cc:2932] T 2ecebe8dad3946bd956afacf695fabaa P 66df61031360493886f1170a809e079a [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, VOTER 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:52.744001 1211 raft_consensus.cc:2932] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, VOTER 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:52.745890 955 catalog_manager.cc:5484] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 reported cstate change: config changed from index 30 to 31, VOTER 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129) evicted. New cstate: current_term: 3 leader_uuid: "b1e9945160714b16b2db91f90c5ccc70" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:52.746197 948 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2ecebe8dad3946bd956afacf695fabaa with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:52.748385 1056 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ecebe8dad3946bd956afacf695fabaa with delete_type TABLET_DATA_TOMBSTONED (TS 77e648af9dbb46fc881340efe1d1e3c3 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:48298
I20240502 07:59:52.748598 1897 tablet_replica.cc:330] T 2ecebe8dad3946bd956afacf695fabaa P 77e648af9dbb46fc881340efe1d1e3c3: stopping tablet replica
I20240502 07:59:52.748740 1897 raft_consensus.cc:2229] T 2ecebe8dad3946bd956afacf695fabaa P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:52.748950 1897 raft_consensus.cc:2258] T 2ecebe8dad3946bd956afacf695fabaa P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:52.749572 1897 ts_tablet_manager.cc:1875] T 2ecebe8dad3946bd956afacf695fabaa P 77e648af9dbb46fc881340efe1d1e3c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:52.751323 1897 ts_tablet_manager.cc:1888] T 2ecebe8dad3946bd956afacf695fabaa P 77e648af9dbb46fc881340efe1d1e3c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.30
I20240502 07:59:52.751397 1897 log.cc:1196] T 2ecebe8dad3946bd956afacf695fabaa P 77e648af9dbb46fc881340efe1d1e3c3: Deleting WAL directory at /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-0-root/wals/2ecebe8dad3946bd956afacf695fabaa
I20240502 07:59:52.751852 949 catalog_manager.cc:4830] TS 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761): tablet 2ecebe8dad3946bd956afacf695fabaa (table test-workload [id=c4044eda63f741838cbc708a8054d3b9]) successfully deleted
I20240502 07:59:52.755077 1530 raft_consensus.cc:3037] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a [term 1 LEARNER]: Advancing to term 2
I20240502 07:59:52.756032 1530 raft_consensus.cc:1261] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a [term 2 LEARNER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 07:59:52.756413 1867 consensus_queue.cc:1035] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:52.758061 1867 raft_consensus.cc:1050] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3: attempting to promote NON_VOTER 66df61031360493886f1170a809e079a to VOTER
W20240502 07:59:52.758139 1867 raft_consensus.cc:1053] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3: Unable to promote non-voter 66df61031360493886f1170a809e079a: Illegal state: Leader has not yet committed an operation in its own term
I20240502 07:59:52.799247 1867 raft_consensus.cc:1050] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3: attempting to promote NON_VOTER eb0a49d2a951470e85adbcd797128e69 to VOTER
I20240502 07:59:52.799624 1867 consensus_queue.cc:237] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.32, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } }
I20240502 07:59:52.801029 1453 raft_consensus.cc:1261] T fda98cfe14f947969fb41e26c75bc384 P 9d994926b3d54e6395a0d60df8a1db07 [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20240502 07:59:52.801029 1139 raft_consensus.cc:1261] T fda98cfe14f947969fb41e26c75bc384 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20240502 07:59:52.801100 1605 raft_consensus.cc:1261] T fda98cfe14f947969fb41e26c75bc384 P eb0a49d2a951470e85adbcd797128e69 [term 1 LEARNER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20240502 07:59:52.801431 1867 consensus_queue.cc:1035] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 07:59:52.801635 1881 consensus_queue.cc:1035] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 07:59:52.801795 1879 consensus_queue.cc:1035] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 07:59:52.804066 1867 raft_consensus.cc:2932] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Committing config change with OpId 1.33: config changed from index 32 to 33, eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:52.804498 1453 raft_consensus.cc:2932] T fda98cfe14f947969fb41e26c75bc384 P 9d994926b3d54e6395a0d60df8a1db07 [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index 32 to 33, eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:52.804718 1605 raft_consensus.cc:2932] T fda98cfe14f947969fb41e26c75bc384 P eb0a49d2a951470e85adbcd797128e69 [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index 32 to 33, eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:52.806576 962 catalog_manager.cc:5484] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 reported cstate change: config changed from index 32 to 33, eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "77e648af9dbb46fc881340efe1d1e3c3" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:52.806986 1139 raft_consensus.cc:2932] T fda98cfe14f947969fb41e26c75bc384 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index 32 to 33, eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:52.810554 1066 consensus_queue.cc:237] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 1.33, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } }
I20240502 07:59:52.812285 1453 raft_consensus.cc:1261] T fda98cfe14f947969fb41e26c75bc384 P 9d994926b3d54e6395a0d60df8a1db07 [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 1 index: 34. (index mismatch)
I20240502 07:59:52.812287 1605 raft_consensus.cc:1261] T fda98cfe14f947969fb41e26c75bc384 P eb0a49d2a951470e85adbcd797128e69 [term 1 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 1 index: 34. (index mismatch)
I20240502 07:59:52.812724 1881 consensus_queue.cc:1035] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 07:59:52.812933 1725 consensus_queue.cc:1035] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 07:59:52.814435 1867 raft_consensus.cc:2932] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Committing config change with OpId 1.34: config changed from index 33 to 34, VOTER b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:52.814990 1453 raft_consensus.cc:2932] T fda98cfe14f947969fb41e26c75bc384 P 9d994926b3d54e6395a0d60df8a1db07 [term 1 FOLLOWER]: Committing config change with OpId 1.34: config changed from index 33 to 34, VOTER b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:52.814990 1605 raft_consensus.cc:2932] T fda98cfe14f947969fb41e26c75bc384 P eb0a49d2a951470e85adbcd797128e69 [term 1 FOLLOWER]: Committing config change with OpId 1.34: config changed from index 33 to 34, VOTER b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:52.816170 949 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fda98cfe14f947969fb41e26c75bc384 with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:52.816880 955 catalog_manager.cc:5484] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 reported cstate change: config changed from index 33 to 34, VOTER b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130) evicted. New cstate: current_term: 1 leader_uuid: "77e648af9dbb46fc881340efe1d1e3c3" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:52.819406 1127 tablet_service.cc:1514] Processing DeleteTablet for tablet fda98cfe14f947969fb41e26c75bc384 with delete_type TABLET_DATA_TOMBSTONED (TS b1e9945160714b16b2db91f90c5ccc70 not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:47876
I20240502 07:59:52.819904 1908 tablet_replica.cc:330] T fda98cfe14f947969fb41e26c75bc384 P b1e9945160714b16b2db91f90c5ccc70: stopping tablet replica
I20240502 07:59:52.820201 1908 raft_consensus.cc:2229] T fda98cfe14f947969fb41e26c75bc384 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:52.820457 1908 raft_consensus.cc:2258] T fda98cfe14f947969fb41e26c75bc384 P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:52.821337 1908 ts_tablet_manager.cc:1875] T fda98cfe14f947969fb41e26c75bc384 P b1e9945160714b16b2db91f90c5ccc70: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:52.823328 1908 ts_tablet_manager.cc:1888] T fda98cfe14f947969fb41e26c75bc384 P b1e9945160714b16b2db91f90c5ccc70: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.33
I20240502 07:59:52.823406 1908 log.cc:1196] T fda98cfe14f947969fb41e26c75bc384 P b1e9945160714b16b2db91f90c5ccc70: Deleting WAL directory at /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-1-root/wals/fda98cfe14f947969fb41e26c75bc384
I20240502 07:59:52.823962 948 catalog_manager.cc:4830] TS b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203): tablet fda98cfe14f947969fb41e26c75bc384 (table test-workload [id=c4044eda63f741838cbc708a8054d3b9]) successfully deleted
I20240502 07:59:52.835866 1211 raft_consensus.cc:1261] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 07:59:52.836246 1867 consensus_queue.cc:1035] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:52.838959 1139 raft_consensus.cc:1261] T 58c5f52a420c467684b2cb50972e7c99 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 07:59:52.839330 1879 consensus_queue.cc:1035] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:52.840229 1879 raft_consensus.cc:1050] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3: attempting to promote NON_VOTER 66df61031360493886f1170a809e079a to VOTER
I20240502 07:59:52.840641 1879 consensus_queue.cc:237] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 29, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } }
I20240502 07:59:52.841873 1139 raft_consensus.cc:1261] T 58c5f52a420c467684b2cb50972e7c99 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 07:59:52.842185 1530 raft_consensus.cc:1261] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a [term 2 LEARNER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 07:59:52.842718 1725 consensus_queue.cc:1035] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 07:59:52.842986 1725 consensus_queue.cc:1035] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 07:59:52.843312 1211 raft_consensus.cc:1261] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 07:59:52.844728 1881 raft_consensus.cc:2932] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 29 to 31, 66df61031360493886f1170a809e079a (127.29.106.133) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:52.844926 1867 consensus_queue.cc:1035] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 07:59:52.845108 1530 raft_consensus.cc:2932] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, 66df61031360493886f1170a809e079a (127.29.106.133) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:52.847041 1139 raft_consensus.cc:2932] T 58c5f52a420c467684b2cb50972e7c99 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, 66df61031360493886f1170a809e079a (127.29.106.133) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:52.847450 955 catalog_manager.cc:5484] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 reported cstate change: config changed from index 29 to 31, 66df61031360493886f1170a809e079a (127.29.106.133) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "77e648af9dbb46fc881340efe1d1e3c3" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:52.847532 1211 raft_consensus.cc:2932] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, 66df61031360493886f1170a809e079a (127.29.106.133) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:53.268991 1066 raft_consensus.cc:1261] T 5437f94679a841e3980053e69d624d76 P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 07:59:53.269445 1772 consensus_queue.cc:1035] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:53.272388 1211 raft_consensus.cc:1261] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 07:59:53.272670 1605 raft_consensus.cc:3037] T 5437f94679a841e3980053e69d624d76 P eb0a49d2a951470e85adbcd797128e69 [term 1 LEARNER]: Advancing to term 3
I20240502 07:59:53.273802 1605 raft_consensus.cc:1261] T 5437f94679a841e3980053e69d624d76 P eb0a49d2a951470e85adbcd797128e69 [term 3 LEARNER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 07:59:53.274152 1914 consensus_queue.cc:1035] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 07:59:53.274384 1913 consensus_queue.cc:1035] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:53.276337 1914 raft_consensus.cc:1050] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70: attempting to promote NON_VOTER eb0a49d2a951470e85adbcd797128e69 to VOTER
I20240502 07:59:53.277328 1914 consensus_queue.cc:237] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 30, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } }
I20240502 07:59:53.278506 1605 raft_consensus.cc:1261] T 5437f94679a841e3980053e69d624d76 P eb0a49d2a951470e85adbcd797128e69 [term 3 LEARNER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240502 07:59:53.278506 1211 raft_consensus.cc:1261] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240502 07:59:53.278725 1066 raft_consensus.cc:1261] T 5437f94679a841e3980053e69d624d76 P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240502 07:59:53.279031 1914 consensus_queue.cc:1035] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 07:59:53.279142 1918 consensus_queue.cc:1035] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 07:59:53.279273 1913 consensus_queue.cc:1035] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 07:59:53.280897 1913 raft_consensus.cc:2932] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 29 to 32, eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:53.281284 1211 raft_consensus.cc:2932] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 29 to 32, eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:53.283552 1063 raft_consensus.cc:2932] T 5437f94679a841e3980053e69d624d76 P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 29 to 32, eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:53.283849 960 catalog_manager.cc:5484] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f reported cstate change: config changed from index 29 to 32, eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "b1e9945160714b16b2db91f90c5ccc70" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:53.286298 1605 raft_consensus.cc:2932] T 5437f94679a841e3980053e69d624d76 P eb0a49d2a951470e85adbcd797128e69 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 29 to 32, eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:53.286748 1139 consensus_queue.cc:237] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } }
I20240502 07:59:53.287931 1066 raft_consensus.cc:1261] T 5437f94679a841e3980053e69d624d76 P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240502 07:59:53.288059 1605 raft_consensus.cc:1261] T 5437f94679a841e3980053e69d624d76 P eb0a49d2a951470e85adbcd797128e69 [term 3 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240502 07:59:53.288425 1918 consensus_queue.cc:1035] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 07:59:53.288846 1919 consensus_queue.cc:1035] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 07:59:53.290431 1919 raft_consensus.cc:2932] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [term 3 LEADER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:53.290997 1063 raft_consensus.cc:2932] T 5437f94679a841e3980053e69d624d76 P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:53.291262 1605 raft_consensus.cc:2932] T 5437f94679a841e3980053e69d624d76 P eb0a49d2a951470e85adbcd797128e69 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:53.292333 948 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5437f94679a841e3980053e69d624d76 with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:53.293071 960 catalog_manager.cc:5484] T 5437f94679a841e3980053e69d624d76 P 77e648af9dbb46fc881340efe1d1e3c3 reported cstate change: config changed from index 32 to 33, VOTER 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131) evicted. New cstate: current_term: 3 leader_uuid: "b1e9945160714b16b2db91f90c5ccc70" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:53.295624 1207 tablet_service.cc:1514] Processing DeleteTablet for tablet 5437f94679a841e3980053e69d624d76 with delete_type TABLET_DATA_TOMBSTONED (TS 27ec1b26c2f64cc682ed7ae12e133a9f not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:52036
I20240502 07:59:53.296150 1920 tablet_replica.cc:330] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f: stopping tablet replica
I20240502 07:59:53.296312 1920 raft_consensus.cc:2229] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:53.296499 1920 raft_consensus.cc:2258] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:53.297247 1920 ts_tablet_manager.cc:1875] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:53.299230 1920 ts_tablet_manager.cc:1888] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.32
I20240502 07:59:53.299293 1920 log.cc:1196] T 5437f94679a841e3980053e69d624d76 P 27ec1b26c2f64cc682ed7ae12e133a9f: Deleting WAL directory at /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-2-root/wals/5437f94679a841e3980053e69d624d76
I20240502 07:59:53.299757 951 catalog_manager.cc:4830] TS 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737): tablet 5437f94679a841e3980053e69d624d76 (table test-workload [id=c4044eda63f741838cbc708a8054d3b9]) successfully deleted
I20240502 07:59:53.415597 1066 consensus_queue.cc:237] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } }
I20240502 07:59:53.416713 1530 raft_consensus.cc:1261] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a [term 2 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240502 07:59:53.416759 1211 raft_consensus.cc:1261] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Refusing update from remote peer 77e648af9dbb46fc881340efe1d1e3c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240502 07:59:53.417073 1725 consensus_queue.cc:1035] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 07:59:53.417253 1881 consensus_queue.cc:1035] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 07:59:53.418892 1879 raft_consensus.cc:2932] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:53.419430 1530 raft_consensus.cc:2932] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:53.420804 949 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 58c5f52a420c467684b2cb50972e7c99 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:53.421272 1211 raft_consensus.cc:2932] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:53.421566 961 catalog_manager.cc:5484] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 reported cstate change: config changed from index 31 to 32, VOTER b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130) evicted. New cstate: current_term: 2 leader_uuid: "77e648af9dbb46fc881340efe1d1e3c3" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:53.424037 1127 tablet_service.cc:1514] Processing DeleteTablet for tablet 58c5f52a420c467684b2cb50972e7c99 with delete_type TABLET_DATA_TOMBSTONED (TS b1e9945160714b16b2db91f90c5ccc70 not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:47876
I20240502 07:59:53.424448 1921 tablet_replica.cc:330] T 58c5f52a420c467684b2cb50972e7c99 P b1e9945160714b16b2db91f90c5ccc70: stopping tablet replica
I20240502 07:59:53.424611 1921 raft_consensus.cc:2229] T 58c5f52a420c467684b2cb50972e7c99 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:53.424782 1921 raft_consensus.cc:2258] T 58c5f52a420c467684b2cb50972e7c99 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:53.425453 1921 ts_tablet_manager.cc:1875] T 58c5f52a420c467684b2cb50972e7c99 P b1e9945160714b16b2db91f90c5ccc70: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:53.427330 1921 ts_tablet_manager.cc:1888] T 58c5f52a420c467684b2cb50972e7c99 P b1e9945160714b16b2db91f90c5ccc70: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.31
I20240502 07:59:53.427402 1921 log.cc:1196] T 58c5f52a420c467684b2cb50972e7c99 P b1e9945160714b16b2db91f90c5ccc70: Deleting WAL directory at /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-1-root/wals/58c5f52a420c467684b2cb50972e7c99
I20240502 07:59:53.427953 948 catalog_manager.cc:4830] TS b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203): tablet 58c5f52a420c467684b2cb50972e7c99 (table test-workload [id=c4044eda63f741838cbc708a8054d3b9]) successfully deleted
I20240502 07:59:53.940992 1018 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:59:53.942991 1063 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "58c5f52a420c467684b2cb50972e7c99"
dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3"
mode: GRACEFUL
new_leader_uuid: "66df61031360493886f1170a809e079a"
from {username='slave'} at 127.0.0.1:48332
I20240502 07:59:53.943151 1063 raft_consensus.cc:595] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 LEADER]: Received request to transfer leadership to TS 66df61031360493886f1170a809e079a
I20240502 07:59:53.944087 1139 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cdf0ff2967c040a589f915172d3f446b"
dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
mode: GRACEFUL
new_leader_uuid: "9d994926b3d54e6395a0d60df8a1db07"
from {username='slave'} at 127.0.0.1:47914
I20240502 07:59:53.944197 1139 raft_consensus.cc:595] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Received request to transfer leadership to TS 9d994926b3d54e6395a0d60df8a1db07
I20240502 07:59:53.944994 1063 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0d982dd52d246348fb3984fd1c3a40f"
dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3"
mode: GRACEFUL
new_leader_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
from {username='slave'} at 127.0.0.1:48332
I20240502 07:59:53.945102 1063 raft_consensus.cc:595] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 LEADER]: Received request to transfer leadership to TS 27ec1b26c2f64cc682ed7ae12e133a9f
I20240502 07:59:53.945530 1018 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240502 07:59:53.945701 1018 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:59:54.115643 1918 raft_consensus.cc:979] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70: : Instructing follower 9d994926b3d54e6395a0d60df8a1db07 to start an election
I20240502 07:59:54.115779 1918 raft_consensus.cc:1067] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Signalling peer 9d994926b3d54e6395a0d60df8a1db07 to start an election
I20240502 07:59:54.116394 1453 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cdf0ff2967c040a589f915172d3f446b"
dest_uuid: "9d994926b3d54e6395a0d60df8a1db07"
from {username='slave'} at 127.0.0.1:41168
I20240502 07:59:54.116523 1453 raft_consensus.cc:482] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:54.116564 1453 raft_consensus.cc:3037] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:54.117421 1453 raft_consensus.cc:504] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } }
I20240502 07:59:54.117738 1453 leader_election.cc:288] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07 [CANDIDATE]: Term 2 election: Requested vote from peers b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203), 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:54.118068 1139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf0ff2967c040a589f915172d3f446b" candidate_uuid: "9d994926b3d54e6395a0d60df8a1db07" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
I20240502 07:59:54.118106 1211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf0ff2967c040a589f915172d3f446b" candidate_uuid: "9d994926b3d54e6395a0d60df8a1db07" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
I20240502 07:59:54.118180 1139 raft_consensus.cc:3032] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:54.118186 1211 raft_consensus.cc:3037] T cdf0ff2967c040a589f915172d3f446b P 27ec1b26c2f64cc682ed7ae12e133a9f [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:54.118248 1139 raft_consensus.cc:729] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b1e9945160714b16b2db91f90c5ccc70, State: Running, Role: LEADER
I20240502 07:59:54.118428 1139 consensus_queue.cc:260] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } }
I20240502 07:59:54.118634 1139 raft_consensus.cc:3037] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:54.119045 1211 raft_consensus.cc:2454] T cdf0ff2967c040a589f915172d3f446b P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d994926b3d54e6395a0d60df8a1db07 in term 2.
I20240502 07:59:54.119331 1420 leader_election.cc:302] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ec1b26c2f64cc682ed7ae12e133a9f, 9d994926b3d54e6395a0d60df8a1db07; no voters:
I20240502 07:59:54.119488 1846 raft_consensus.cc:2781] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:54.119642 1846 raft_consensus.cc:686] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07 [term 2 LEADER]: Becoming Leader. State: Replica: 9d994926b3d54e6395a0d60df8a1db07, State: Running, Role: LEADER
I20240502 07:59:54.119768 1139 raft_consensus.cc:2454] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d994926b3d54e6395a0d60df8a1db07 in term 2.
I20240502 07:59:54.119781 1846 consensus_queue.cc:237] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } }
I20240502 07:59:54.121398 960 catalog_manager.cc:5484] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07 reported cstate change: term changed from 1 to 2, leader changed from b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130) to 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132). New cstate: current_term: 2 leader_uuid: "9d994926b3d54e6395a0d60df8a1db07" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:54.188897 1879 raft_consensus.cc:979] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3: : Instructing follower 27ec1b26c2f64cc682ed7ae12e133a9f to start an election
I20240502 07:59:54.189000 1879 raft_consensus.cc:1067] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 LEADER]: Signalling peer 27ec1b26c2f64cc682ed7ae12e133a9f to start an election
I20240502 07:59:54.189402 1211 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d0d982dd52d246348fb3984fd1c3a40f"
dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
from {username='slave'} at 127.0.0.1:52060
I20240502 07:59:54.189538 1211 raft_consensus.cc:482] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:54.189610 1211 raft_consensus.cc:3037] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Advancing to term 4
I20240502 07:59:54.190424 1211 raft_consensus.cc:504] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:54.190763 1211 leader_election.cc:288] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [CANDIDATE]: Term 4 election: Requested vote from peers 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761), b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203)
I20240502 07:59:54.191071 1063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d982dd52d246348fb3984fd1c3a40f" candidate_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3"
I20240502 07:59:54.191188 1063 raft_consensus.cc:3032] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 07:59:54.191076 1143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d982dd52d246348fb3984fd1c3a40f" candidate_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
I20240502 07:59:54.191246 1063 raft_consensus.cc:729] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77e648af9dbb46fc881340efe1d1e3c3, State: Running, Role: LEADER
I20240502 07:59:54.191305 1143 raft_consensus.cc:3037] T d0d982dd52d246348fb3984fd1c3a40f P b1e9945160714b16b2db91f90c5ccc70 [term 3 FOLLOWER]: Advancing to term 4
I20240502 07:59:54.191401 1063 consensus_queue.cc:260] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:54.191563 1063 raft_consensus.cc:3037] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 FOLLOWER]: Advancing to term 4
I20240502 07:59:54.192179 1143 raft_consensus.cc:2454] T d0d982dd52d246348fb3984fd1c3a40f P b1e9945160714b16b2db91f90c5ccc70 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27ec1b26c2f64cc682ed7ae12e133a9f in term 4.
I20240502 07:59:54.192241 1063 raft_consensus.cc:2454] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27ec1b26c2f64cc682ed7ae12e133a9f in term 4.
I20240502 07:59:54.192514 1181 leader_election.cc:302] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ec1b26c2f64cc682ed7ae12e133a9f, b1e9945160714b16b2db91f90c5ccc70; no voters:
I20240502 07:59:54.192698 1916 raft_consensus.cc:2781] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [term 4 FOLLOWER]: Leader election won for term 4
I20240502 07:59:54.192917 1916 raft_consensus.cc:686] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [term 4 LEADER]: Becoming Leader. State: Replica: 27ec1b26c2f64cc682ed7ae12e133a9f, State: Running, Role: LEADER
I20240502 07:59:54.193058 1916 consensus_queue.cc:237] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } }
I20240502 07:59:54.194643 960 catalog_manager.cc:5484] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f reported cstate change: term changed from 3 to 4, leader changed from 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129) to 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131). New cstate: current_term: 4 leader_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: HEALTHY } } }
I20240502 07:59:54.425189 1211 consensus_queue.cc:237] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 22, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } }
W20240502 07:59:54.426798 1184 consensus_peers.cc:485] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f -> Peer eb0a49d2a951470e85adbcd797128e69 (127.29.106.134:34541): Couldn't send request to peer eb0a49d2a951470e85adbcd797128e69. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 682e26b333104f82863b92c545cfdafe. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:54.426905 1143 raft_consensus.cc:1261] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 07:59:54.427098 1063 raft_consensus.cc:1261] T 682e26b333104f82863b92c545cfdafe P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 07:59:54.427320 1932 consensus_queue.cc:1035] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 07:59:54.427520 1916 consensus_queue.cc:1035] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 07:59:54.429214 1932 raft_consensus.cc:2932] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } } }
I20240502 07:59:54.429656 1143 raft_consensus.cc:2932] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } } }
I20240502 07:59:54.429720 1063 raft_consensus.cc:2932] T 682e26b333104f82863b92c545cfdafe P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } } }
I20240502 07:59:54.431861 960 catalog_manager.cc:5484] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 reported cstate change: config changed from index -1 to 29, NON_VOTER eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) added. New cstate: current_term: 2 leader_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } } }
I20240502 07:59:54.432451 1143 consensus_queue.cc:237] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } }
I20240502 07:59:54.433764 1530 raft_consensus.cc:1261] T 2ecebe8dad3946bd956afacf695fabaa P 66df61031360493886f1170a809e079a [term 3 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
W20240502 07:59:54.433948 1106 consensus_peers.cc:485] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 -> Peer 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132:44697): Couldn't send request to peer 9d994926b3d54e6395a0d60df8a1db07. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ecebe8dad3946bd956afacf695fabaa. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:54.434000 1211 raft_consensus.cc:1261] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240502 07:59:54.434142 1936 consensus_queue.cc:1035] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 07:59:54.434341 1918 consensus_queue.cc:1035] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 07:59:54.436168 1918 raft_consensus.cc:2932] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 31 to 32, NON_VOTER 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } } }
I20240502 07:59:54.436507 1530 raft_consensus.cc:2932] T 2ecebe8dad3946bd956afacf695fabaa P 66df61031360493886f1170a809e079a [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, NON_VOTER 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } } }
I20240502 07:59:54.438220 1211 raft_consensus.cc:2932] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, NON_VOTER 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } } }
I20240502 07:59:54.438372 962 catalog_manager.cc:5484] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 reported cstate change: config changed from index 31 to 32, NON_VOTER 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) added. New cstate: current_term: 3 leader_uuid: "b1e9945160714b16b2db91f90c5ccc70" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:54.443650 1453 consensus_queue.cc:237] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } }
I20240502 07:59:54.445060 1143 raft_consensus.cc:1261] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Refusing update from remote peer 9d994926b3d54e6395a0d60df8a1db07: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240502 07:59:54.445394 1211 raft_consensus.cc:1261] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Refusing update from remote peer 9d994926b3d54e6395a0d60df8a1db07: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240502 07:59:54.445415 1942 consensus_queue.cc:1035] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 07:59:54.445726 1923 consensus_queue.cc:1035] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 07:59:54.447836 1942 raft_consensus.cc:2932] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 31 to 33, NON_VOTER 66df61031360493886f1170a809e079a (127.29.106.133) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } } }
I20240502 07:59:54.448877 1211 raft_consensus.cc:2932] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 31 to 33, NON_VOTER 66df61031360493886f1170a809e079a (127.29.106.133) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } } }
W20240502 07:59:54.449661 1420 consensus_peers.cc:485] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 -> Peer 66df61031360493886f1170a809e079a (127.29.106.133:38717): Couldn't send request to peer 66df61031360493886f1170a809e079a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c1b6523d6724a02acc9ae19f3f0255e. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:54.449749 1143 raft_consensus.cc:2932] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 31 to 33, NON_VOTER 66df61031360493886f1170a809e079a (127.29.106.133) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } } }
I20240502 07:59:54.451548 1211 consensus_queue.cc:237] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } }
I20240502 07:59:54.452101 960 catalog_manager.cc:5484] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 reported cstate change: config changed from index 31 to 33, NON_VOTER 66df61031360493886f1170a809e079a (127.29.106.133) added. New cstate: current_term: 2 leader_uuid: "9d994926b3d54e6395a0d60df8a1db07" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:54.453537 1879 raft_consensus.cc:979] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3: : Instructing follower 66df61031360493886f1170a809e079a to start an election
I20240502 07:59:54.453634 1879 raft_consensus.cc:1067] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 LEADER]: Signalling peer 66df61031360493886f1170a809e079a to start an election
I20240502 07:59:54.454008 1530 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "58c5f52a420c467684b2cb50972e7c99"
dest_uuid: "66df61031360493886f1170a809e079a"
from {username='slave'} at 127.0.0.1:47864
I20240502 07:59:54.454114 1530 raft_consensus.cc:482] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:54.454169 1530 raft_consensus.cc:3037] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a [term 2 FOLLOWER]: Advancing to term 3
I20240502 07:59:54.454777 1143 raft_consensus.cc:1261] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240502 07:59:54.455185 1530 raft_consensus.cc:504] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } }
I20240502 07:59:54.455420 1916 consensus_queue.cc:1035] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 07:59:54.455607 1530 leader_election.cc:288] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a [CANDIDATE]: Term 3 election: Requested vote from peers 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761), 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:54.455849 1063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c5f52a420c467684b2cb50972e7c99" candidate_uuid: "66df61031360493886f1170a809e079a" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: true dest_uuid: "77e648af9dbb46fc881340efe1d1e3c3"
I20240502 07:59:54.455942 1063 raft_consensus.cc:3032] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 07:59:54.455978 1063 raft_consensus.cc:729] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 77e648af9dbb46fc881340efe1d1e3c3, State: Running, Role: LEADER
I20240502 07:59:54.455988 1211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c5f52a420c467684b2cb50972e7c99" candidate_uuid: "66df61031360493886f1170a809e079a" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: true dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
I20240502 07:59:54.456082 1211 raft_consensus.cc:3037] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Advancing to term 3
I20240502 07:59:54.456096 1063 consensus_queue.cc:260] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } }
I20240502 07:59:54.456243 1063 raft_consensus.cc:3037] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Advancing to term 3
I20240502 07:59:54.455194 1605 raft_consensus.cc:1261] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [term 2 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240502 07:59:54.457055 1211 raft_consensus.cc:2454] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66df61031360493886f1170a809e079a in term 3.
I20240502 07:59:54.457233 1063 raft_consensus.cc:2454] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66df61031360493886f1170a809e079a in term 3.
I20240502 07:59:54.457377 1916 consensus_queue.cc:1035] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 07:59:54.457427 1496 leader_election.cc:302] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ec1b26c2f64cc682ed7ae12e133a9f, 66df61031360493886f1170a809e079a; no voters:
I20240502 07:59:54.457571 1857 raft_consensus.cc:2781] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a [term 3 FOLLOWER]: Leader election won for term 3
I20240502 07:59:54.457739 1857 raft_consensus.cc:686] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a [term 3 LEADER]: Becoming Leader. State: Replica: 66df61031360493886f1170a809e079a, State: Running, Role: LEADER
I20240502 07:59:54.457873 1857 consensus_queue.cc:237] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } }
I20240502 07:59:54.458964 1932 raft_consensus.cc:2932] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 31 to 33, NON_VOTER 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } } }
I20240502 07:59:54.459348 1605 raft_consensus.cc:2932] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 31 to 33, NON_VOTER 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } } }
I20240502 07:59:54.460981 962 catalog_manager.cc:5484] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a reported cstate change: term changed from 2 to 3, leader changed from 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129) to 66df61031360493886f1170a809e079a (127.29.106.133). New cstate: current_term: 3 leader_uuid: "66df61031360493886f1170a809e079a" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:54.461069 1143 raft_consensus.cc:2932] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 31 to 33, NON_VOTER 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } } }
W20240502 07:59:54.461375 1181 consensus_peers.cc:485] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f -> Peer 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132:44697): Couldn't send request to peer 9d994926b3d54e6395a0d60df8a1db07. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6edf17550e3f4a0b945477edc6de6af1. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:54.462325 960 catalog_manager.cc:5484] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 reported cstate change: config changed from index 31 to 33, NON_VOTER 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) added. New cstate: current_term: 2 leader_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } } }
I20240502 07:59:54.466837 1530 consensus_queue.cc:237] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } }
I20240502 07:59:54.468333 1063 raft_consensus.cc:1261] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Refusing update from remote peer 66df61031360493886f1170a809e079a: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240502 07:59:54.468344 1143 raft_consensus.cc:1261] T df4233b549b34b01b6f9662c7e7b3515 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Refusing update from remote peer 66df61031360493886f1170a809e079a: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240502 07:59:54.468822 1955 consensus_queue.cc:1035] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 07:59:54.469230 1955 consensus_queue.cc:1035] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 07:59:54.470419 1857 raft_consensus.cc:2932] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 31 to 33, NON_VOTER eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } } }
W20240502 07:59:54.472076 1496 consensus_peers.cc:485] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a -> Peer eb0a49d2a951470e85adbcd797128e69 (127.29.106.134:34541): Couldn't send request to peer eb0a49d2a951470e85adbcd797128e69. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df4233b549b34b01b6f9662c7e7b3515. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:54.472564 1063 raft_consensus.cc:2932] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 31 to 33, NON_VOTER eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } } }
I20240502 07:59:54.472736 955 catalog_manager.cc:5484] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a reported cstate change: config changed from index 31 to 33, NON_VOTER eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) added. New cstate: current_term: 2 leader_uuid: "66df61031360493886f1170a809e079a" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:54.473469 1143 raft_consensus.cc:2932] T df4233b549b34b01b6f9662c7e7b3515 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 31 to 33, NON_VOTER eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } } }
I20240502 07:59:54.477147 1605 consensus_queue.cc:237] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 32, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } }
I20240502 07:59:54.478638 1063 raft_consensus.cc:1261] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 FOLLOWER]: Refusing update from remote peer eb0a49d2a951470e85adbcd797128e69: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20240502 07:59:54.478930 1211 raft_consensus.cc:1261] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Refusing update from remote peer eb0a49d2a951470e85adbcd797128e69: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20240502 07:59:54.479024 1961 consensus_queue.cc:1035] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 07:59:54.479272 1859 consensus_queue.cc:1035] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 07:59:54.480947 1962 raft_consensus.cc:2932] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [term 3 LEADER]: Committing config change with OpId 3.34: config changed from index 32 to 34, NON_VOTER 66df61031360493886f1170a809e079a (127.29.106.133) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } } }
I20240502 07:59:54.481329 1063 raft_consensus.cc:2932] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 32 to 34, NON_VOTER 66df61031360493886f1170a809e079a (127.29.106.133) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } } }
I20240502 07:59:54.481400 1211 raft_consensus.cc:2932] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 32 to 34, NON_VOTER 66df61031360493886f1170a809e079a (127.29.106.133) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } } }
I20240502 07:59:54.483704 955 catalog_manager.cc:5484] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f reported cstate change: config changed from index 32 to 34, NON_VOTER 66df61031360493886f1170a809e079a (127.29.106.133) added. New cstate: current_term: 3 leader_uuid: "eb0a49d2a951470e85adbcd797128e69" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } } }
W20240502 07:59:54.484308 1571 consensus_peers.cc:485] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 -> Peer 66df61031360493886f1170a809e079a (127.29.106.133:38717): Couldn't send request to peer 66df61031360493886f1170a809e079a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69a9ebd0c390436b8ace08089f719358. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:54.502710 1211 raft_consensus.cc:1261] T cdf0ff2967c040a589f915172d3f446b P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Refusing update from remote peer 9d994926b3d54e6395a0d60df8a1db07: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 07:59:54.503136 1846 consensus_queue.cc:1035] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 07:59:54.506147 1143 raft_consensus.cc:1261] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Refusing update from remote peer 9d994926b3d54e6395a0d60df8a1db07: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 07:59:54.506528 1923 consensus_queue.cc:1035] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 07:59:54.642004 1143 raft_consensus.cc:1261] T d0d982dd52d246348fb3984fd1c3a40f P b1e9945160714b16b2db91f90c5ccc70 [term 4 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240502 07:59:54.642410 1929 consensus_queue.cc:1035] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 07:59:54.644613 1063 raft_consensus.cc:1261] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 4 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240502 07:59:54.645303 1929 consensus_queue.cc:1035] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 07:59:54.822788 1974 ts_tablet_manager.cc:902] T 2c1b6523d6724a02acc9ae19f3f0255e P 66df61031360493886f1170a809e079a: Initiating tablet copy from peer 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132:44697)
I20240502 07:59:54.823436 1974 tablet_copy_client.cc:308] T 2c1b6523d6724a02acc9ae19f3f0255e P 66df61031360493886f1170a809e079a: tablet copy: Beginning tablet copy session from remote peer at address 127.29.106.132:44697
I20240502 07:59:54.827257 1464 tablet_copy_service.cc:140] P 9d994926b3d54e6395a0d60df8a1db07: Received BeginTabletCopySession request for tablet 2c1b6523d6724a02acc9ae19f3f0255e from peer 66df61031360493886f1170a809e079a ({username='slave'} at 127.0.0.1:49556)
I20240502 07:59:54.827390 1464 tablet_copy_service.cc:161] P 9d994926b3d54e6395a0d60df8a1db07: Beginning new tablet copy session on tablet 2c1b6523d6724a02acc9ae19f3f0255e from peer 66df61031360493886f1170a809e079a at {username='slave'} at 127.0.0.1:49556: session id = 66df61031360493886f1170a809e079a-2c1b6523d6724a02acc9ae19f3f0255e
I20240502 07:59:54.828390 1464 tablet_copy_source_session.cc:205] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:54.829157 1974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c1b6523d6724a02acc9ae19f3f0255e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:54.831198 1974 tablet_copy_client.cc:783] T 2c1b6523d6724a02acc9ae19f3f0255e P 66df61031360493886f1170a809e079a: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:54.831350 1974 tablet_copy_client.cc:647] T 2c1b6523d6724a02acc9ae19f3f0255e P 66df61031360493886f1170a809e079a: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:54.832458 1978 tablet_copy_client.cc:1002] T 2c1b6523d6724a02acc9ae19f3f0255e P 66df61031360493886f1170a809e079a: tablet copy: Injecting latency into file download: 2000
I20240502 07:59:54.875788 1979 ts_tablet_manager.cc:902] T 69a9ebd0c390436b8ace08089f719358 P 66df61031360493886f1170a809e079a: Initiating tablet copy from peer eb0a49d2a951470e85adbcd797128e69 (127.29.106.134:34541)
I20240502 07:59:54.876441 1979 tablet_copy_client.cc:308] T 69a9ebd0c390436b8ace08089f719358 P 66df61031360493886f1170a809e079a: tablet copy: Beginning tablet copy session from remote peer at address 127.29.106.134:34541
I20240502 07:59:54.876864 1615 tablet_copy_service.cc:140] P eb0a49d2a951470e85adbcd797128e69: Received BeginTabletCopySession request for tablet 69a9ebd0c390436b8ace08089f719358 from peer 66df61031360493886f1170a809e079a ({username='slave'} at 127.0.0.1:48368)
I20240502 07:59:54.876950 1615 tablet_copy_service.cc:161] P eb0a49d2a951470e85adbcd797128e69: Beginning new tablet copy session on tablet 69a9ebd0c390436b8ace08089f719358 from peer 66df61031360493886f1170a809e079a at {username='slave'} at 127.0.0.1:48368: session id = 66df61031360493886f1170a809e079a-69a9ebd0c390436b8ace08089f719358
I20240502 07:59:54.877801 1615 tablet_copy_source_session.cc:205] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:54.878593 1979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69a9ebd0c390436b8ace08089f719358. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:54.880937 1979 tablet_copy_client.cc:783] T 69a9ebd0c390436b8ace08089f719358 P 66df61031360493886f1170a809e079a: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:54.881105 1979 tablet_copy_client.cc:647] T 69a9ebd0c390436b8ace08089f719358 P 66df61031360493886f1170a809e079a: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:54.881517 1984 ts_tablet_manager.cc:902] T 2ecebe8dad3946bd956afacf695fabaa P 9d994926b3d54e6395a0d60df8a1db07: Initiating tablet copy from peer b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203)
I20240502 07:59:54.882077 1984 tablet_copy_client.cc:308] T 2ecebe8dad3946bd956afacf695fabaa P 9d994926b3d54e6395a0d60df8a1db07: tablet copy: Beginning tablet copy session from remote peer at address 127.29.106.130:38203
I20240502 07:59:54.882294 1983 tablet_copy_client.cc:1002] T 69a9ebd0c390436b8ace08089f719358 P 66df61031360493886f1170a809e079a: tablet copy: Injecting latency into file download: 2000
I20240502 07:59:54.882457 1153 tablet_copy_service.cc:140] P b1e9945160714b16b2db91f90c5ccc70: Received BeginTabletCopySession request for tablet 2ecebe8dad3946bd956afacf695fabaa from peer 9d994926b3d54e6395a0d60df8a1db07 ({username='slave'} at 127.0.0.1:47922)
I20240502 07:59:54.882544 1153 tablet_copy_service.cc:161] P b1e9945160714b16b2db91f90c5ccc70: Beginning new tablet copy session on tablet 2ecebe8dad3946bd956afacf695fabaa from peer 9d994926b3d54e6395a0d60df8a1db07 at {username='slave'} at 127.0.0.1:47922: session id = 9d994926b3d54e6395a0d60df8a1db07-2ecebe8dad3946bd956afacf695fabaa
I20240502 07:59:54.883400 1153 tablet_copy_source_session.cc:205] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:54.884223 1984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ecebe8dad3946bd956afacf695fabaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:54.887286 1984 tablet_copy_client.cc:783] T 2ecebe8dad3946bd956afacf695fabaa P 9d994926b3d54e6395a0d60df8a1db07: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:54.887454 1984 tablet_copy_client.cc:647] T 2ecebe8dad3946bd956afacf695fabaa P 9d994926b3d54e6395a0d60df8a1db07: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:54.888566 1985 tablet_copy_client.cc:1002] T 2ecebe8dad3946bd956afacf695fabaa P 9d994926b3d54e6395a0d60df8a1db07: tablet copy: Injecting latency into file download: 2000
I20240502 07:59:54.892513 1989 ts_tablet_manager.cc:902] T 682e26b333104f82863b92c545cfdafe P eb0a49d2a951470e85adbcd797128e69: Initiating tablet copy from peer 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:54.893146 1989 tablet_copy_client.cc:308] T 682e26b333104f82863b92c545cfdafe P eb0a49d2a951470e85adbcd797128e69: tablet copy: Beginning tablet copy session from remote peer at address 127.29.106.131:36737
I20240502 07:59:54.893555 1228 tablet_copy_service.cc:140] P 27ec1b26c2f64cc682ed7ae12e133a9f: Received BeginTabletCopySession request for tablet 682e26b333104f82863b92c545cfdafe from peer eb0a49d2a951470e85adbcd797128e69 ({username='slave'} at 127.0.0.1:52082)
I20240502 07:59:54.893678 1228 tablet_copy_service.cc:161] P 27ec1b26c2f64cc682ed7ae12e133a9f: Beginning new tablet copy session on tablet 682e26b333104f82863b92c545cfdafe from peer eb0a49d2a951470e85adbcd797128e69 at {username='slave'} at 127.0.0.1:52082: session id = eb0a49d2a951470e85adbcd797128e69-682e26b333104f82863b92c545cfdafe
I20240502 07:59:54.894731 1228 tablet_copy_source_session.cc:205] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:54.895519 1989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 682e26b333104f82863b92c545cfdafe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:54.897918 1989 tablet_copy_client.cc:783] T 682e26b333104f82863b92c545cfdafe P eb0a49d2a951470e85adbcd797128e69: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:54.898092 1989 tablet_copy_client.cc:647] T 682e26b333104f82863b92c545cfdafe P eb0a49d2a951470e85adbcd797128e69: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:54.898981 1993 tablet_copy_client.cc:1002] T 682e26b333104f82863b92c545cfdafe P eb0a49d2a951470e85adbcd797128e69: tablet copy: Injecting latency into file download: 2000
I20240502 07:59:54.925274 1211 raft_consensus.cc:1261] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Refusing update from remote peer 66df61031360493886f1170a809e079a: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240502 07:59:54.925734 1857 consensus_queue.cc:1035] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 07:59:54.928182 1063 raft_consensus.cc:1261] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 FOLLOWER]: Refusing update from remote peer 66df61031360493886f1170a809e079a: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240502 07:59:54.928575 1857 consensus_queue.cc:1035] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 07:59:55.011929 1997 ts_tablet_manager.cc:902] T 6edf17550e3f4a0b945477edc6de6af1 P 9d994926b3d54e6395a0d60df8a1db07: Initiating tablet copy from peer 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737)
I20240502 07:59:55.012490 1997 tablet_copy_client.cc:308] T 6edf17550e3f4a0b945477edc6de6af1 P 9d994926b3d54e6395a0d60df8a1db07: tablet copy: Beginning tablet copy session from remote peer at address 127.29.106.131:36737
I20240502 07:59:55.012884 1228 tablet_copy_service.cc:140] P 27ec1b26c2f64cc682ed7ae12e133a9f: Received BeginTabletCopySession request for tablet 6edf17550e3f4a0b945477edc6de6af1 from peer 9d994926b3d54e6395a0d60df8a1db07 ({username='slave'} at 127.0.0.1:40320)
I20240502 07:59:55.013008 1228 tablet_copy_service.cc:161] P 27ec1b26c2f64cc682ed7ae12e133a9f: Beginning new tablet copy session on tablet 6edf17550e3f4a0b945477edc6de6af1 from peer 9d994926b3d54e6395a0d60df8a1db07 at {username='slave'} at 127.0.0.1:40320: session id = 9d994926b3d54e6395a0d60df8a1db07-6edf17550e3f4a0b945477edc6de6af1
I20240502 07:59:55.013757 1228 tablet_copy_source_session.cc:205] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:55.014496 1997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6edf17550e3f4a0b945477edc6de6af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:55.016597 1997 tablet_copy_client.cc:783] T 6edf17550e3f4a0b945477edc6de6af1 P 9d994926b3d54e6395a0d60df8a1db07: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:55.016844 1997 tablet_copy_client.cc:647] T 6edf17550e3f4a0b945477edc6de6af1 P 9d994926b3d54e6395a0d60df8a1db07: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:55.017894 2001 tablet_copy_client.cc:1002] T 6edf17550e3f4a0b945477edc6de6af1 P 9d994926b3d54e6395a0d60df8a1db07: tablet copy: Injecting latency into file download: 2000
I20240502 07:59:55.090803 2004 ts_tablet_manager.cc:902] T df4233b549b34b01b6f9662c7e7b3515 P eb0a49d2a951470e85adbcd797128e69: Initiating tablet copy from peer 66df61031360493886f1170a809e079a (127.29.106.133:38717)
I20240502 07:59:55.091382 2004 tablet_copy_client.cc:308] T df4233b549b34b01b6f9662c7e7b3515 P eb0a49d2a951470e85adbcd797128e69: tablet copy: Beginning tablet copy session from remote peer at address 127.29.106.133:38717
I20240502 07:59:55.091785 1540 tablet_copy_service.cc:140] P 66df61031360493886f1170a809e079a: Received BeginTabletCopySession request for tablet df4233b549b34b01b6f9662c7e7b3515 from peer eb0a49d2a951470e85adbcd797128e69 ({username='slave'} at 127.0.0.1:56196)
I20240502 07:59:55.091893 1540 tablet_copy_service.cc:161] P 66df61031360493886f1170a809e079a: Beginning new tablet copy session on tablet df4233b549b34b01b6f9662c7e7b3515 from peer eb0a49d2a951470e85adbcd797128e69 at {username='slave'} at 127.0.0.1:56196: session id = eb0a49d2a951470e85adbcd797128e69-df4233b549b34b01b6f9662c7e7b3515
I20240502 07:59:55.092952 1540 tablet_copy_source_session.cc:205] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:55.093674 2004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df4233b549b34b01b6f9662c7e7b3515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:55.095678 2004 tablet_copy_client.cc:783] T df4233b549b34b01b6f9662c7e7b3515 P eb0a49d2a951470e85adbcd797128e69: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:55.095834 2004 tablet_copy_client.cc:647] T df4233b549b34b01b6f9662c7e7b3515 P eb0a49d2a951470e85adbcd797128e69: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:55.096982 2008 tablet_copy_client.cc:1002] T df4233b549b34b01b6f9662c7e7b3515 P eb0a49d2a951470e85adbcd797128e69: tablet copy: Injecting latency into file download: 2000
I20240502 07:59:55.945976 1018 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:59:55.947739 1218 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6edf17550e3f4a0b945477edc6de6af1"
dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
mode: GRACEFUL
new_leader_uuid: "b1e9945160714b16b2db91f90c5ccc70"
from {username='slave'} at 127.0.0.1:40314
I20240502 07:59:55.947886 1218 raft_consensus.cc:595] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Received request to transfer leadership to TS b1e9945160714b16b2db91f90c5ccc70
I20240502 07:59:55.948244 1018 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240502 07:59:55.948393 1018 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:59:55.994468 1916 raft_consensus.cc:979] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f: : Instructing follower b1e9945160714b16b2db91f90c5ccc70 to start an election
I20240502 07:59:55.994625 1916 raft_consensus.cc:1067] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Signalling peer b1e9945160714b16b2db91f90c5ccc70 to start an election
I20240502 07:59:55.995102 1139 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6edf17550e3f4a0b945477edc6de6af1"
dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
from {username='slave'} at 127.0.0.1:47886
I20240502 07:59:55.995225 1139 raft_consensus.cc:482] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:55.995290 1139 raft_consensus.cc:3037] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Advancing to term 3
I20240502 07:59:55.996558 1139 raft_consensus.cc:504] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } }
I20240502 07:59:55.996961 1139 leader_election.cc:288] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [CANDIDATE]: Term 3 election: Requested vote from peers 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737), eb0a49d2a951470e85adbcd797128e69 (127.29.106.134:34541)
I20240502 07:59:55.997303 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6edf17550e3f4a0b945477edc6de6af1" candidate_uuid: "b1e9945160714b16b2db91f90c5ccc70" candidate_term: 3 candidate_status { last_received { term: 2 index: 33 } } ignore_live_leader: true dest_uuid: "eb0a49d2a951470e85adbcd797128e69"
I20240502 07:59:55.997313 1218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6edf17550e3f4a0b945477edc6de6af1" candidate_uuid: "b1e9945160714b16b2db91f90c5ccc70" candidate_term: 3 candidate_status { last_received { term: 2 index: 33 } } ignore_live_leader: true dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
I20240502 07:59:55.997422 1604 raft_consensus.cc:3037] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [term 2 FOLLOWER]: Advancing to term 3
I20240502 07:59:55.997463 1218 raft_consensus.cc:3032] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Stepping down as leader of term 2
I20240502 07:59:55.997504 1218 raft_consensus.cc:729] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 27ec1b26c2f64cc682ed7ae12e133a9f, State: Running, Role: LEADER
I20240502 07:59:55.997697 1218 consensus_queue.cc:260] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 33, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } }
I20240502 07:59:55.998018 1218 raft_consensus.cc:3037] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Advancing to term 3
I20240502 07:59:55.998275 1604 raft_consensus.cc:2454] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1e9945160714b16b2db91f90c5ccc70 in term 3.
I20240502 07:59:55.998531 1109 leader_election.cc:302] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1e9945160714b16b2db91f90c5ccc70, eb0a49d2a951470e85adbcd797128e69; no voters:
I20240502 07:59:55.998714 2019 raft_consensus.cc:2781] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 07:59:55.998904 2019 raft_consensus.cc:686] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 3 LEADER]: Becoming Leader. State: Replica: b1e9945160714b16b2db91f90c5ccc70, State: Running, Role: LEADER
I20240502 07:59:55.999042 2019 consensus_queue.cc:237] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 33, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } }
I20240502 07:59:55.999118 1218 raft_consensus.cc:2454] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1e9945160714b16b2db91f90c5ccc70 in term 3.
I20240502 07:59:56.000743 962 catalog_manager.cc:5484] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 reported cstate change: term changed from 2 to 3, leader changed from 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131) to b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130). New cstate: current_term: 3 leader_uuid: "b1e9945160714b16b2db91f90c5ccc70" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:56.004961 1139 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6edf17550e3f4a0b945477edc6de6af1"
dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47876
I20240502 07:59:56.005146 1139 raft_consensus.cc:595] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 3 LEADER]: Received request to transfer leadership
I20240502 07:59:56.010690 1139 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6edf17550e3f4a0b945477edc6de6af1"
dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47876
I20240502 07:59:56.010834 1139 raft_consensus.cc:595] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 3 LEADER]: Received request to transfer leadership
I20240502 07:59:56.011757 1139 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6edf17550e3f4a0b945477edc6de6af1"
dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47876
I20240502 07:59:56.011878 1139 raft_consensus.cc:595] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 3 LEADER]: Received request to transfer leadership
W20240502 07:59:56.012522 1013 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 6edf17550e3f4a0b945477edc6de6af1 already in progress
W20240502 07:59:56.012588 1013 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 6edf17550e3f4a0b945477edc6de6af1 already in progress
I20240502 07:59:56.578965 1211 raft_consensus.cc:1261] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20240502 07:59:56.579372 2019 consensus_queue.cc:1035] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 07:59:56.580946 2028 raft_consensus.cc:979] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70: : Instructing follower 27ec1b26c2f64cc682ed7ae12e133a9f to start an election
I20240502 07:59:56.581055 2019 raft_consensus.cc:1067] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 3 LEADER]: Signalling peer 27ec1b26c2f64cc682ed7ae12e133a9f to start an election
I20240502 07:59:56.582185 1605 raft_consensus.cc:1261] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [term 3 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20240502 07:59:56.582271 1218 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6edf17550e3f4a0b945477edc6de6af1"
dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
from {username='slave'} at 127.0.0.1:52044
I20240502 07:59:56.582377 1218 raft_consensus.cc:482] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:56.582442 1218 raft_consensus.cc:3037] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Advancing to term 4
I20240502 07:59:56.583061 2028 consensus_queue.cc:1035] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 07:59:56.583464 1218 raft_consensus.cc:504] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } }
I20240502 07:59:56.583875 1218 leader_election.cc:288] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [CANDIDATE]: Term 4 election: Requested vote from peers b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203), eb0a49d2a951470e85adbcd797128e69 (127.29.106.134:34541)
I20240502 07:59:56.584180 1143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6edf17550e3f4a0b945477edc6de6af1" candidate_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" candidate_term: 4 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: true dest_uuid: "b1e9945160714b16b2db91f90c5ccc70"
I20240502 07:59:56.584304 1143 raft_consensus.cc:3032] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 07:59:56.584347 1143 raft_consensus.cc:729] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b1e9945160714b16b2db91f90c5ccc70, State: Running, Role: LEADER
W20240502 07:59:56.584376 1106 consensus_peers.cc:485] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 -> Peer 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132:44697): Couldn't send request to peer 9d994926b3d54e6395a0d60df8a1db07. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:56.584478 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6edf17550e3f4a0b945477edc6de6af1" candidate_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" candidate_term: 4 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: true dest_uuid: "eb0a49d2a951470e85adbcd797128e69"
I20240502 07:59:56.584467 1143 consensus_queue.cc:260] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 3.34, Last appended by leader: 34, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } }
I20240502 07:59:56.584654 1143 raft_consensus.cc:3037] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 3 FOLLOWER]: Advancing to term 4
I20240502 07:59:56.586010 1143 raft_consensus.cc:2454] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27ec1b26c2f64cc682ed7ae12e133a9f in term 4.
I20240502 07:59:56.586277 1181 leader_election.cc:302] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27ec1b26c2f64cc682ed7ae12e133a9f, b1e9945160714b16b2db91f90c5ccc70; no voters: eb0a49d2a951470e85adbcd797128e69
I20240502 07:59:56.586429 1916 raft_consensus.cc:2781] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 4 FOLLOWER]: Leader election won for term 4
I20240502 07:59:56.586643 1916 raft_consensus.cc:686] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 4 LEADER]: Becoming Leader. State: Replica: 27ec1b26c2f64cc682ed7ae12e133a9f, State: Running, Role: LEADER
I20240502 07:59:56.586802 1916 consensus_queue.cc:237] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 3.34, Last appended by leader: 34, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } }
I20240502 07:59:56.588145 961 catalog_manager.cc:5484] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f reported cstate change: term changed from 3 to 4, leader changed from b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130) to 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131). New cstate: current_term: 4 leader_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:56.832791 1974 tablet_copy_client.cc:523] T 2c1b6523d6724a02acc9ae19f3f0255e P 66df61031360493886f1170a809e079a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:56.834563 1974 tablet_bootstrap.cc:492] T 2c1b6523d6724a02acc9ae19f3f0255e P 66df61031360493886f1170a809e079a: Bootstrap starting.
I20240502 07:59:56.861403 1974 tablet_bootstrap.cc:492] T 2c1b6523d6724a02acc9ae19f3f0255e P 66df61031360493886f1170a809e079a: Bootstrap replayed 1/1 log segments. Stats: ops{read=33 overwritten=0 applied=33 ignored=0} inserts{seen=136 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:56.861737 1974 tablet_bootstrap.cc:492] T 2c1b6523d6724a02acc9ae19f3f0255e P 66df61031360493886f1170a809e079a: Bootstrap complete.
I20240502 07:59:56.861939 1974 ts_tablet_manager.cc:1367] T 2c1b6523d6724a02acc9ae19f3f0255e P 66df61031360493886f1170a809e079a: Time spent bootstrapping tablet: real 0.027s user 0.018s sys 0.009s
I20240502 07:59:56.862360 1974 raft_consensus.cc:348] T 2c1b6523d6724a02acc9ae19f3f0255e P 66df61031360493886f1170a809e079a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } }
I20240502 07:59:56.862470 1974 raft_consensus.cc:729] T 2c1b6523d6724a02acc9ae19f3f0255e P 66df61031360493886f1170a809e079a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 66df61031360493886f1170a809e079a, State: Initialized, Role: LEARNER
I20240502 07:59:56.862582 1974 consensus_queue.cc:260] T 2c1b6523d6724a02acc9ae19f3f0255e P 66df61031360493886f1170a809e079a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 33, Last appended: 2.33, Last appended by leader: 33, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } }
I20240502 07:59:56.862859 1974 ts_tablet_manager.cc:1398] T 2c1b6523d6724a02acc9ae19f3f0255e P 66df61031360493886f1170a809e079a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:56.863284 1464 tablet_copy_service.cc:342] P 9d994926b3d54e6395a0d60df8a1db07: Request end of tablet copy session 66df61031360493886f1170a809e079a-2c1b6523d6724a02acc9ae19f3f0255e received from {username='slave'} at 127.0.0.1:49556
I20240502 07:59:56.863358 1464 tablet_copy_service.cc:433] P 9d994926b3d54e6395a0d60df8a1db07: ending tablet copy session 66df61031360493886f1170a809e079a-2c1b6523d6724a02acc9ae19f3f0255e on tablet 2c1b6523d6724a02acc9ae19f3f0255e with peer 66df61031360493886f1170a809e079a
I20240502 07:59:56.882747 1979 tablet_copy_client.cc:523] T 69a9ebd0c390436b8ace08089f719358 P 66df61031360493886f1170a809e079a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:56.884398 1979 tablet_bootstrap.cc:492] T 69a9ebd0c390436b8ace08089f719358 P 66df61031360493886f1170a809e079a: Bootstrap starting.
I20240502 07:59:56.889087 1984 tablet_copy_client.cc:523] T 2ecebe8dad3946bd956afacf695fabaa P 9d994926b3d54e6395a0d60df8a1db07: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:56.891522 1984 tablet_bootstrap.cc:492] T 2ecebe8dad3946bd956afacf695fabaa P 9d994926b3d54e6395a0d60df8a1db07: Bootstrap starting.
I20240502 07:59:56.899255 1989 tablet_copy_client.cc:523] T 682e26b333104f82863b92c545cfdafe P eb0a49d2a951470e85adbcd797128e69: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:56.900568 1989 tablet_bootstrap.cc:492] T 682e26b333104f82863b92c545cfdafe P eb0a49d2a951470e85adbcd797128e69: Bootstrap starting.
I20240502 07:59:56.914847 1984 tablet_bootstrap.cc:492] T 2ecebe8dad3946bd956afacf695fabaa P 9d994926b3d54e6395a0d60df8a1db07: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=92 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:56.915196 1984 tablet_bootstrap.cc:492] T 2ecebe8dad3946bd956afacf695fabaa P 9d994926b3d54e6395a0d60df8a1db07: Bootstrap complete.
I20240502 07:59:56.915406 1984 ts_tablet_manager.cc:1367] T 2ecebe8dad3946bd956afacf695fabaa P 9d994926b3d54e6395a0d60df8a1db07: Time spent bootstrapping tablet: real 0.024s user 0.022s sys 0.000s
I20240502 07:59:56.915926 1984 raft_consensus.cc:348] T 2ecebe8dad3946bd956afacf695fabaa P 9d994926b3d54e6395a0d60df8a1db07 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } }
I20240502 07:59:56.916340 1984 raft_consensus.cc:729] T 2ecebe8dad3946bd956afacf695fabaa P 9d994926b3d54e6395a0d60df8a1db07 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 9d994926b3d54e6395a0d60df8a1db07, State: Initialized, Role: LEARNER
I20240502 07:59:56.916450 1984 consensus_queue.cc:260] T 2ecebe8dad3946bd956afacf695fabaa P 9d994926b3d54e6395a0d60df8a1db07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } }
I20240502 07:59:56.916743 1984 ts_tablet_manager.cc:1398] T 2ecebe8dad3946bd956afacf695fabaa P 9d994926b3d54e6395a0d60df8a1db07: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 07:59:56.920351 1153 tablet_copy_service.cc:342] P b1e9945160714b16b2db91f90c5ccc70: Request end of tablet copy session 9d994926b3d54e6395a0d60df8a1db07-2ecebe8dad3946bd956afacf695fabaa received from {username='slave'} at 127.0.0.1:47922
I20240502 07:59:56.920449 1153 tablet_copy_service.cc:433] P b1e9945160714b16b2db91f90c5ccc70: ending tablet copy session 9d994926b3d54e6395a0d60df8a1db07-2ecebe8dad3946bd956afacf695fabaa on tablet 2ecebe8dad3946bd956afacf695fabaa with peer 9d994926b3d54e6395a0d60df8a1db07
I20240502 07:59:56.925242 1979 tablet_bootstrap.cc:492] T 69a9ebd0c390436b8ace08089f719358 P 66df61031360493886f1170a809e079a: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=118 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:56.925526 1979 tablet_bootstrap.cc:492] T 69a9ebd0c390436b8ace08089f719358 P 66df61031360493886f1170a809e079a: Bootstrap complete.
I20240502 07:59:56.925720 1979 ts_tablet_manager.cc:1367] T 69a9ebd0c390436b8ace08089f719358 P 66df61031360493886f1170a809e079a: Time spent bootstrapping tablet: real 0.041s user 0.023s sys 0.005s
I20240502 07:59:56.926177 1979 raft_consensus.cc:348] T 69a9ebd0c390436b8ace08089f719358 P 66df61031360493886f1170a809e079a [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } }
I20240502 07:59:56.926280 1979 raft_consensus.cc:729] T 69a9ebd0c390436b8ace08089f719358 P 66df61031360493886f1170a809e079a [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 66df61031360493886f1170a809e079a, State: Initialized, Role: LEARNER
I20240502 07:59:56.926347 1979 consensus_queue.cc:260] T 69a9ebd0c390436b8ace08089f719358 P 66df61031360493886f1170a809e079a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 3.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: NON_VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: true } }
I20240502 07:59:56.926599 1979 ts_tablet_manager.cc:1398] T 69a9ebd0c390436b8ace08089f719358 P 66df61031360493886f1170a809e079a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:56.927162 1615 tablet_copy_service.cc:342] P eb0a49d2a951470e85adbcd797128e69: Request end of tablet copy session 66df61031360493886f1170a809e079a-69a9ebd0c390436b8ace08089f719358 received from {username='slave'} at 127.0.0.1:48368
I20240502 07:59:56.927253 1615 tablet_copy_service.cc:433] P eb0a49d2a951470e85adbcd797128e69: ending tablet copy session 66df61031360493886f1170a809e079a-69a9ebd0c390436b8ace08089f719358 on tablet 69a9ebd0c390436b8ace08089f719358 with peer 66df61031360493886f1170a809e079a
I20240502 07:59:56.936069 1989 tablet_bootstrap.cc:492] T 682e26b333104f82863b92c545cfdafe P eb0a49d2a951470e85adbcd797128e69: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:56.936437 1989 tablet_bootstrap.cc:492] T 682e26b333104f82863b92c545cfdafe P eb0a49d2a951470e85adbcd797128e69: Bootstrap complete.
I20240502 07:59:56.936648 1989 ts_tablet_manager.cc:1367] T 682e26b333104f82863b92c545cfdafe P eb0a49d2a951470e85adbcd797128e69: Time spent bootstrapping tablet: real 0.036s user 0.018s sys 0.007s
I20240502 07:59:56.937139 1989 raft_consensus.cc:348] T 682e26b333104f82863b92c545cfdafe P eb0a49d2a951470e85adbcd797128e69 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } }
I20240502 07:59:56.937253 1989 raft_consensus.cc:729] T 682e26b333104f82863b92c545cfdafe P eb0a49d2a951470e85adbcd797128e69 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: eb0a49d2a951470e85adbcd797128e69, State: Initialized, Role: LEARNER
I20240502 07:59:56.937346 1989 consensus_queue.cc:260] T 682e26b333104f82863b92c545cfdafe P eb0a49d2a951470e85adbcd797128e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } }
I20240502 07:59:56.937705 1989 ts_tablet_manager.cc:1398] T 682e26b333104f82863b92c545cfdafe P eb0a49d2a951470e85adbcd797128e69: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:56.938095 1228 tablet_copy_service.cc:342] P 27ec1b26c2f64cc682ed7ae12e133a9f: Request end of tablet copy session eb0a49d2a951470e85adbcd797128e69-682e26b333104f82863b92c545cfdafe received from {username='slave'} at 127.0.0.1:52082
I20240502 07:59:56.938175 1228 tablet_copy_service.cc:433] P 27ec1b26c2f64cc682ed7ae12e133a9f: ending tablet copy session eb0a49d2a951470e85adbcd797128e69-682e26b333104f82863b92c545cfdafe on tablet 682e26b333104f82863b92c545cfdafe with peer eb0a49d2a951470e85adbcd797128e69
I20240502 07:59:57.018177 1997 tablet_copy_client.cc:523] T 6edf17550e3f4a0b945477edc6de6af1 P 9d994926b3d54e6395a0d60df8a1db07: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:57.019750 1997 tablet_bootstrap.cc:492] T 6edf17550e3f4a0b945477edc6de6af1 P 9d994926b3d54e6395a0d60df8a1db07: Bootstrap starting.
I20240502 07:59:57.036620 1143 raft_consensus.cc:1261] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 4 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 3 index: 34. Preceding OpId from leader: term: 4 index: 35. (index mismatch)
I20240502 07:59:57.037020 1916 consensus_queue.cc:1035] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240502 07:59:57.039278 1604 raft_consensus.cc:3037] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [term 3 FOLLOWER]: Advancing to term 4
I20240502 07:59:57.040253 1604 raft_consensus.cc:1261] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [term 4 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 3 index: 34. Preceding OpId from leader: term: 4 index: 35. (index mismatch)
I20240502 07:59:57.040678 2035 consensus_queue.cc:1035] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
W20240502 07:59:57.042794 1181 consensus_peers.cc:485] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f -> Peer 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132:44697): Couldn't send request to peer 9d994926b3d54e6395a0d60df8a1db07. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:57.046073 1997 tablet_bootstrap.cc:492] T 6edf17550e3f4a0b945477edc6de6af1 P 9d994926b3d54e6395a0d60df8a1db07: Bootstrap replayed 1/1 log segments. Stats: ops{read=33 overwritten=0 applied=33 ignored=0} inserts{seen=110 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:57.046422 1997 tablet_bootstrap.cc:492] T 6edf17550e3f4a0b945477edc6de6af1 P 9d994926b3d54e6395a0d60df8a1db07: Bootstrap complete.
I20240502 07:59:57.046613 1997 ts_tablet_manager.cc:1367] T 6edf17550e3f4a0b945477edc6de6af1 P 9d994926b3d54e6395a0d60df8a1db07: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.008s
I20240502 07:59:57.047294 1997 raft_consensus.cc:348] T 6edf17550e3f4a0b945477edc6de6af1 P 9d994926b3d54e6395a0d60df8a1db07 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } }
I20240502 07:59:57.047436 1997 raft_consensus.cc:729] T 6edf17550e3f4a0b945477edc6de6af1 P 9d994926b3d54e6395a0d60df8a1db07 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 9d994926b3d54e6395a0d60df8a1db07, State: Initialized, Role: LEARNER
I20240502 07:59:57.047544 1997 consensus_queue.cc:260] T 6edf17550e3f4a0b945477edc6de6af1 P 9d994926b3d54e6395a0d60df8a1db07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 33, Last appended: 2.33, Last appended by leader: 33, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true } }
I20240502 07:59:57.047837 1997 ts_tablet_manager.cc:1398] T 6edf17550e3f4a0b945477edc6de6af1 P 9d994926b3d54e6395a0d60df8a1db07: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:57.048293 1228 tablet_copy_service.cc:342] P 27ec1b26c2f64cc682ed7ae12e133a9f: Request end of tablet copy session 9d994926b3d54e6395a0d60df8a1db07-6edf17550e3f4a0b945477edc6de6af1 received from {username='slave'} at 127.0.0.1:40320
I20240502 07:59:57.048377 1228 tablet_copy_service.cc:433] P 27ec1b26c2f64cc682ed7ae12e133a9f: ending tablet copy session 9d994926b3d54e6395a0d60df8a1db07-6edf17550e3f4a0b945477edc6de6af1 on tablet 6edf17550e3f4a0b945477edc6de6af1 with peer 9d994926b3d54e6395a0d60df8a1db07
I20240502 07:59:57.060410 1604 raft_consensus.cc:1203] T 682e26b333104f82863b92c545cfdafe P eb0a49d2a951470e85adbcd797128e69 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.28->[2.29-2.29] Dedup: 2.29->[]
I20240502 07:59:57.097288 2004 tablet_copy_client.cc:523] T df4233b549b34b01b6f9662c7e7b3515 P eb0a49d2a951470e85adbcd797128e69: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:57.099004 2004 tablet_bootstrap.cc:492] T df4233b549b34b01b6f9662c7e7b3515 P eb0a49d2a951470e85adbcd797128e69: Bootstrap starting.
I20240502 07:59:57.125957 2004 tablet_bootstrap.cc:492] T df4233b549b34b01b6f9662c7e7b3515 P eb0a49d2a951470e85adbcd797128e69: Bootstrap replayed 1/1 log segments. Stats: ops{read=33 overwritten=0 applied=33 ignored=0} inserts{seen=126 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:57.126343 2004 tablet_bootstrap.cc:492] T df4233b549b34b01b6f9662c7e7b3515 P eb0a49d2a951470e85adbcd797128e69: Bootstrap complete.
I20240502 07:59:57.126603 2004 ts_tablet_manager.cc:1367] T df4233b549b34b01b6f9662c7e7b3515 P eb0a49d2a951470e85adbcd797128e69: Time spent bootstrapping tablet: real 0.028s user 0.017s sys 0.010s
I20240502 07:59:57.127135 2004 raft_consensus.cc:348] T df4233b549b34b01b6f9662c7e7b3515 P eb0a49d2a951470e85adbcd797128e69 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } }
I20240502 07:59:57.127264 2004 raft_consensus.cc:729] T df4233b549b34b01b6f9662c7e7b3515 P eb0a49d2a951470e85adbcd797128e69 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: eb0a49d2a951470e85adbcd797128e69, State: Initialized, Role: LEARNER
I20240502 07:59:57.127379 2004 consensus_queue.cc:260] T df4233b549b34b01b6f9662c7e7b3515 P eb0a49d2a951470e85adbcd797128e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 33, Last appended: 2.33, Last appended by leader: 33, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: NON_VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: true } }
I20240502 07:59:57.128216 2004 ts_tablet_manager.cc:1398] T df4233b549b34b01b6f9662c7e7b3515 P eb0a49d2a951470e85adbcd797128e69: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 07:59:57.128729 1540 tablet_copy_service.cc:342] P 66df61031360493886f1170a809e079a: Request end of tablet copy session eb0a49d2a951470e85adbcd797128e69-df4233b549b34b01b6f9662c7e7b3515 received from {username='slave'} at 127.0.0.1:56196
I20240502 07:59:57.128816 1540 tablet_copy_service.cc:433] P 66df61031360493886f1170a809e079a: ending tablet copy session eb0a49d2a951470e85adbcd797128e69-df4233b549b34b01b6f9662c7e7b3515 on tablet df4233b549b34b01b6f9662c7e7b3515 with peer eb0a49d2a951470e85adbcd797128e69
I20240502 07:59:57.149170 1604 raft_consensus.cc:1203] T df4233b549b34b01b6f9662c7e7b3515 P eb0a49d2a951470e85adbcd797128e69 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.32->[2.33-2.33] Dedup: 2.33->[]
I20240502 07:59:57.195374 1454 raft_consensus.cc:1203] T 2ecebe8dad3946bd956afacf695fabaa P 9d994926b3d54e6395a0d60df8a1db07 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.31->[3.32-3.32] Dedup: 3.32->[]
I20240502 07:59:57.309695 1530 raft_consensus.cc:1203] T 69a9ebd0c390436b8ace08089f719358 P 66df61031360493886f1170a809e079a [term 3 LEARNER]: Deduplicated request from leader. Original: 3.33->[3.34-3.34] Dedup: 3.34->[]
I20240502 07:59:57.378465 1530 raft_consensus.cc:1203] T 2c1b6523d6724a02acc9ae19f3f0255e P 66df61031360493886f1170a809e079a [term 2 LEARNER]: Deduplicated request from leader. Original: 2.32->[2.33-2.33] Dedup: 2.33->[]
I20240502 07:59:57.468746 1454 raft_consensus.cc:3037] T 6edf17550e3f4a0b945477edc6de6af1 P 9d994926b3d54e6395a0d60df8a1db07 [term 2 LEARNER]: Advancing to term 4
I20240502 07:59:57.469730 1454 raft_consensus.cc:1261] T 6edf17550e3f4a0b945477edc6de6af1 P 9d994926b3d54e6395a0d60df8a1db07 [term 4 LEARNER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20240502 07:59:57.470140 2044 consensus_queue.cc:1037] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 07:59:57.472005 2044 raft_consensus.cc:1050] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f: attempting to promote NON_VOTER 9d994926b3d54e6395a0d60df8a1db07 to VOTER
I20240502 07:59:57.472417 2044 consensus_queue.cc:237] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 4.35, Last appended by leader: 34, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } }
I20240502 07:59:57.473470 1454 raft_consensus.cc:1261] T 6edf17550e3f4a0b945477edc6de6af1 P 9d994926b3d54e6395a0d60df8a1db07 [term 4 LEARNER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 4 index: 35. Preceding OpId from leader: term: 4 index: 36. (index mismatch)
I20240502 07:59:57.473915 2035 consensus_queue.cc:1035] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20240502 07:59:57.474062 1143 raft_consensus.cc:1261] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 4 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 4 index: 35. Preceding OpId from leader: term: 4 index: 36. (index mismatch)
I20240502 07:59:57.474522 1916 consensus_queue.cc:1035] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20240502 07:59:57.475232 1604 raft_consensus.cc:1261] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [term 4 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 4 index: 35. Preceding OpId from leader: term: 4 index: 36. (index mismatch)
I20240502 07:59:57.475726 1916 consensus_queue.cc:1035] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20240502 07:59:57.476624 2035 raft_consensus.cc:2932] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 4 LEADER]: Committing config change with OpId 4.36: config changed from index 33 to 36, 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:57.477057 1454 raft_consensus.cc:2932] T 6edf17550e3f4a0b945477edc6de6af1 P 9d994926b3d54e6395a0d60df8a1db07 [term 4 FOLLOWER]: Committing config change with OpId 4.36: config changed from index 33 to 36, 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:57.478273 1139 raft_consensus.cc:2932] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 4 FOLLOWER]: Committing config change with OpId 4.36: config changed from index 33 to 36, 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:57.479187 1604 raft_consensus.cc:2932] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [term 4 FOLLOWER]: Committing config change with OpId 4.36: config changed from index 33 to 36, 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:57.479344 962 catalog_manager.cc:5484] T 6edf17550e3f4a0b945477edc6de6af1 P 9d994926b3d54e6395a0d60df8a1db07 reported cstate change: config changed from index 33 to 36, 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:57.486335 1218 consensus_queue.cc:237] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 4.36, Last appended by leader: 34, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } }
I20240502 07:59:57.487562 1604 raft_consensus.cc:1261] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [term 4 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 4 index: 36. Preceding OpId from leader: term: 4 index: 37. (index mismatch)
I20240502 07:59:57.487599 1454 raft_consensus.cc:1261] T 6edf17550e3f4a0b945477edc6de6af1 P 9d994926b3d54e6395a0d60df8a1db07 [term 4 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 4 index: 36. Preceding OpId from leader: term: 4 index: 37. (index mismatch)
I20240502 07:59:57.487927 2035 consensus_queue.cc:1035] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20240502 07:59:57.488097 2044 consensus_queue.cc:1035] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20240502 07:59:57.489451 2044 raft_consensus.cc:2932] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 4 LEADER]: Committing config change with OpId 4.37: config changed from index 36 to 37, VOTER b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130) evicted. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:57.490084 1454 raft_consensus.cc:2932] T 6edf17550e3f4a0b945477edc6de6af1 P 9d994926b3d54e6395a0d60df8a1db07 [term 4 FOLLOWER]: Committing config change with OpId 4.37: config changed from index 36 to 37, VOTER b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130) evicted. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:57.490051 1605 raft_consensus.cc:2932] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [term 4 FOLLOWER]: Committing config change with OpId 4.37: config changed from index 36 to 37, VOTER b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130) evicted. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:57.491885 951 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6edf17550e3f4a0b945477edc6de6af1 with cas_config_opid_index 36: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:57.492226 961 catalog_manager.cc:5484] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f reported cstate change: config changed from index 36 to 37, VOTER b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130) evicted. New cstate: current_term: 4 leader_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:57.495246 1127 tablet_service.cc:1514] Processing DeleteTablet for tablet 6edf17550e3f4a0b945477edc6de6af1 with delete_type TABLET_DATA_TOMBSTONED (TS b1e9945160714b16b2db91f90c5ccc70 not found in new config with opid_index 37) from {username='slave'} at 127.0.0.1:47876
I20240502 07:59:57.495685 2053 tablet_replica.cc:330] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70: stopping tablet replica
I20240502 07:59:57.495819 2053 raft_consensus.cc:2229] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:57.495994 2053 raft_consensus.cc:2258] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:57.496762 2053 ts_tablet_manager.cc:1875] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:57.499102 2053 ts_tablet_manager.cc:1888] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.36
I20240502 07:59:57.499193 2053 log.cc:1196] T 6edf17550e3f4a0b945477edc6de6af1 P b1e9945160714b16b2db91f90c5ccc70: Deleting WAL directory at /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-1-root/wals/6edf17550e3f4a0b945477edc6de6af1
I20240502 07:59:57.499666 948 catalog_manager.cc:4830] TS b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203): tablet 6edf17550e3f4a0b945477edc6de6af1 (table test-workload [id=c4044eda63f741838cbc708a8054d3b9]) successfully deleted
I20240502 07:59:57.516979 1916 raft_consensus.cc:1050] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f: attempting to promote NON_VOTER eb0a49d2a951470e85adbcd797128e69 to VOTER
I20240502 07:59:57.517413 1916 consensus_queue.cc:237] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 22, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } }
I20240502 07:59:57.518730 1063 raft_consensus.cc:1261] T 682e26b333104f82863b92c545cfdafe P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 07:59:57.518730 1139 raft_consensus.cc:1261] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 07:59:57.518795 1605 raft_consensus.cc:1261] T 682e26b333104f82863b92c545cfdafe P eb0a49d2a951470e85adbcd797128e69 [term 2 LEARNER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 07:59:57.519203 2044 consensus_queue.cc:1035] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:57.519364 2035 consensus_queue.cc:1035] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:57.519680 2035 consensus_queue.cc:1035] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 07:59:57.521306 2044 raft_consensus.cc:2932] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:57.521734 1605 raft_consensus.cc:2932] T 682e26b333104f82863b92c545cfdafe P eb0a49d2a951470e85adbcd797128e69 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:57.523825 1139 raft_consensus.cc:2932] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:57.523981 1063 raft_consensus.cc:2932] T 682e26b333104f82863b92c545cfdafe P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:57.524098 960 catalog_manager.cc:5484] T 682e26b333104f82863b92c545cfdafe P eb0a49d2a951470e85adbcd797128e69 reported cstate change: config changed from index 29 to 30, eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:57.527019 2026 raft_consensus.cc:1050] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a: attempting to promote NON_VOTER eb0a49d2a951470e85adbcd797128e69 to VOTER
I20240502 07:59:57.527359 2026 consensus_queue.cc:237] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 31, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } }
I20240502 07:59:57.528762 1139 raft_consensus.cc:1261] T df4233b549b34b01b6f9662c7e7b3515 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Refusing update from remote peer 66df61031360493886f1170a809e079a: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240502 07:59:57.528928 1605 raft_consensus.cc:1261] T df4233b549b34b01b6f9662c7e7b3515 P eb0a49d2a951470e85adbcd797128e69 [term 2 LEARNER]: Refusing update from remote peer 66df61031360493886f1170a809e079a: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240502 07:59:57.529109 1063 raft_consensus.cc:1261] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Refusing update from remote peer 66df61031360493886f1170a809e079a: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240502 07:59:57.529135 2062 consensus_queue.cc:1035] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 07:59:57.529457 2026 consensus_queue.cc:1035] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 07:59:57.529790 2026 consensus_queue.cc:1035] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 07:59:57.532161 2059 raft_consensus.cc:2932] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [term 2 LEADER]: Committing config change with OpId 2.34: config changed from index 33 to 34, eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:57.532647 1218 consensus_queue.cc:237] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 22, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } }
I20240502 07:59:57.533644 1605 raft_consensus.cc:1261] T 682e26b333104f82863b92c545cfdafe P eb0a49d2a951470e85adbcd797128e69 [term 2 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 07:59:57.533736 1063 raft_consensus.cc:2932] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:57.533934 1139 raft_consensus.cc:1261] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Refusing update from remote peer 27ec1b26c2f64cc682ed7ae12e133a9f: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 07:59:57.534327 2035 consensus_queue.cc:1035] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 07:59:57.534672 2055 consensus_queue.cc:1035] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 07:59:57.534682 1139 raft_consensus.cc:2932] T df4233b549b34b01b6f9662c7e7b3515 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:57.535135 955 catalog_manager.cc:5484] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a reported cstate change: config changed from index 33 to 34, eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "66df61031360493886f1170a809e079a" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:57.535948 1604 raft_consensus.cc:2932] T df4233b549b34b01b6f9662c7e7b3515 P eb0a49d2a951470e85adbcd797128e69 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, eb0a49d2a951470e85adbcd797128e69 (127.29.106.134) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } attrs { replace: true } } peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:57.539685 2035 raft_consensus.cc:2932] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:57.539930 1604 raft_consensus.cc:2932] T 682e26b333104f82863b92c545cfdafe P eb0a49d2a951470e85adbcd797128e69 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:57.542048 951 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 682e26b333104f82863b92c545cfdafe with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:57.542708 961 catalog_manager.cc:5484] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f reported cstate change: config changed from index 30 to 31, VOTER 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129) evicted. New cstate: current_term: 2 leader_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:57.543447 1530 consensus_queue.cc:237] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 2.34, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } }
I20240502 07:59:57.544754 1139 raft_consensus.cc:1261] T df4233b549b34b01b6f9662c7e7b3515 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Refusing update from remote peer 66df61031360493886f1170a809e079a: Log matching property violated. Preceding OpId in replica: term: 2 index: 34. Preceding OpId from leader: term: 2 index: 35. (index mismatch)
I20240502 07:59:57.545089 2061 consensus_queue.cc:1035] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240502 07:59:57.545150 1139 raft_consensus.cc:2932] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:57.545369 1056 tablet_service.cc:1514] Processing DeleteTablet for tablet 682e26b333104f82863b92c545cfdafe with delete_type TABLET_DATA_TOMBSTONED (TS 77e648af9dbb46fc881340efe1d1e3c3 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:48298
I20240502 07:59:57.545823 2066 tablet_replica.cc:330] T 682e26b333104f82863b92c545cfdafe P 77e648af9dbb46fc881340efe1d1e3c3: stopping tablet replica
I20240502 07:59:57.545960 2066 raft_consensus.cc:2229] T 682e26b333104f82863b92c545cfdafe P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:57.547385 2059 raft_consensus.cc:2932] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [term 2 LEADER]: Committing config change with OpId 2.35: config changed from index 34 to 35, VOTER 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:57.548437 1604 raft_consensus.cc:1261] T df4233b549b34b01b6f9662c7e7b3515 P eb0a49d2a951470e85adbcd797128e69 [term 2 FOLLOWER]: Refusing update from remote peer 66df61031360493886f1170a809e079a: Log matching property violated. Preceding OpId in replica: term: 2 index: 34. Preceding OpId from leader: term: 2 index: 35. (index mismatch)
I20240502 07:59:57.549512 1012 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet df4233b549b34b01b6f9662c7e7b3515 with cas_config_opid_index 34: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:57.549513 2059 consensus_queue.cc:1035] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240502 07:59:57.550112 961 catalog_manager.cc:5484] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a reported cstate change: config changed from index 34 to 35, VOTER 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129) evicted. New cstate: current_term: 2 leader_uuid: "66df61031360493886f1170a809e079a" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:57.546451 2066 raft_consensus.cc:2258] T 682e26b333104f82863b92c545cfdafe P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:57.550582 1139 raft_consensus.cc:2932] T df4233b549b34b01b6f9662c7e7b3515 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Committing config change with OpId 2.35: config changed from index 34 to 35, VOTER 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:57.551009 1604 raft_consensus.cc:2932] T df4233b549b34b01b6f9662c7e7b3515 P eb0a49d2a951470e85adbcd797128e69 [term 2 FOLLOWER]: Committing config change with OpId 2.35: config changed from index 34 to 35, VOTER 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } }
I20240502 07:59:57.551326 2066 ts_tablet_manager.cc:1875] T 682e26b333104f82863b92c545cfdafe P 77e648af9dbb46fc881340efe1d1e3c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:57.553555 2066 ts_tablet_manager.cc:1888] T 682e26b333104f82863b92c545cfdafe P 77e648af9dbb46fc881340efe1d1e3c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.30
I20240502 07:59:57.553650 2066 log.cc:1196] T 682e26b333104f82863b92c545cfdafe P 77e648af9dbb46fc881340efe1d1e3c3: Deleting WAL directory at /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-0-root/wals/682e26b333104f82863b92c545cfdafe
I20240502 07:59:57.554236 949 catalog_manager.cc:4830] TS 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761): tablet 682e26b333104f82863b92c545cfdafe (table test-workload [id=c4044eda63f741838cbc708a8054d3b9]) successfully deleted
I20240502 07:59:57.555024 1056 tablet_service.cc:1514] Processing DeleteTablet for tablet df4233b549b34b01b6f9662c7e7b3515 with delete_type TABLET_DATA_TOMBSTONED (TS 77e648af9dbb46fc881340efe1d1e3c3 not found in new config with opid_index 35) from {username='slave'} at 127.0.0.1:48298
I20240502 07:59:57.555466 2066 tablet_replica.cc:330] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3: stopping tablet replica
I20240502 07:59:57.555624 2066 raft_consensus.cc:2229] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:57.556423 2066 raft_consensus.cc:2258] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:57.557029 2066 ts_tablet_manager.cc:1875] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:57.558604 2066 ts_tablet_manager.cc:1888] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.34
I20240502 07:59:57.558674 2066 log.cc:1196] T df4233b549b34b01b6f9662c7e7b3515 P 77e648af9dbb46fc881340efe1d1e3c3: Deleting WAL directory at /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-0-root/wals/df4233b549b34b01b6f9662c7e7b3515
I20240502 07:59:57.559223 949 catalog_manager.cc:4830] TS 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761): tablet df4233b549b34b01b6f9662c7e7b3515 (table test-workload [id=c4044eda63f741838cbc708a8054d3b9]) successfully deleted
I20240502 07:59:57.719915 2067 raft_consensus.cc:1050] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70: attempting to promote NON_VOTER 9d994926b3d54e6395a0d60df8a1db07 to VOTER
I20240502 07:59:57.720937 2067 consensus_queue.cc:237] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 28, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } }
I20240502 07:59:57.722335 1529 raft_consensus.cc:1261] T 2ecebe8dad3946bd956afacf695fabaa P 66df61031360493886f1170a809e079a [term 3 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240502 07:59:57.722831 1453 raft_consensus.cc:1261] T 2ecebe8dad3946bd956afacf695fabaa P 9d994926b3d54e6395a0d60df8a1db07 [term 3 LEARNER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240502 07:59:57.723045 1218 raft_consensus.cc:1261] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240502 07:59:57.723135 2028 consensus_queue.cc:1035] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 07:59:57.723405 2069 consensus_queue.cc:1035] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 07:59:57.723812 2067 consensus_queue.cc:1035] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 07:59:57.725732 2069 raft_consensus.cc:2932] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 3 LEADER]: Committing config change with OpId 3.33: config changed from index 32 to 33, 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:57.727545 1529 raft_consensus.cc:2932] T 2ecebe8dad3946bd956afacf695fabaa P 66df61031360493886f1170a809e079a [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:57.728117 1454 raft_consensus.cc:2932] T 2ecebe8dad3946bd956afacf695fabaa P 9d994926b3d54e6395a0d60df8a1db07 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:57.728966 960 catalog_manager.cc:5484] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 reported cstate change: config changed from index 32 to 33, 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "b1e9945160714b16b2db91f90c5ccc70" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:57.729631 1218 raft_consensus.cc:2932] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:57.731848 1143 consensus_queue.cc:237] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } }
I20240502 07:59:57.733059 1529 raft_consensus.cc:1261] T 2ecebe8dad3946bd956afacf695fabaa P 66df61031360493886f1170a809e079a [term 3 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20240502 07:59:57.733059 1453 raft_consensus.cc:1261] T 2ecebe8dad3946bd956afacf695fabaa P 9d994926b3d54e6395a0d60df8a1db07 [term 3 FOLLOWER]: Refusing update from remote peer b1e9945160714b16b2db91f90c5ccc70: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20240502 07:59:57.733508 2074 consensus_queue.cc:1035] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 07:59:57.733876 2067 consensus_queue.cc:1035] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 07:59:57.735421 2069 raft_consensus.cc:2932] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 3 LEADER]: Committing config change with OpId 3.34: config changed from index 33 to 34, VOTER 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:57.735862 1453 raft_consensus.cc:2932] T 2ecebe8dad3946bd956afacf695fabaa P 9d994926b3d54e6395a0d60df8a1db07 [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 33 to 34, VOTER 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:57.735946 1529 raft_consensus.cc:2932] T 2ecebe8dad3946bd956afacf695fabaa P 66df61031360493886f1170a809e079a [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 33 to 34, VOTER 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:57.738088 962 catalog_manager.cc:5484] T 2ecebe8dad3946bd956afacf695fabaa P 66df61031360493886f1170a809e079a reported cstate change: config changed from index 33 to 34, VOTER 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131) evicted. New cstate: current_term: 3 leader_uuid: "b1e9945160714b16b2db91f90c5ccc70" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } }
I20240502 07:59:57.738306 948 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2ecebe8dad3946bd956afacf695fabaa with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:57.740650 1207 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ecebe8dad3946bd956afacf695fabaa with delete_type TABLET_DATA_TOMBSTONED (TS 27ec1b26c2f64cc682ed7ae12e133a9f not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:52036
I20240502 07:59:57.741179 2075 tablet_replica.cc:330] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f: stopping tablet replica
I20240502 07:59:57.741550 2075 raft_consensus.cc:2229] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:57.741778 2075 raft_consensus.cc:2258] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:57.742442 2075 ts_tablet_manager.cc:1875] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:57.744686 2075 ts_tablet_manager.cc:1888] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.33
I20240502 07:59:57.744763 2075 log.cc:1196] T 2ecebe8dad3946bd956afacf695fabaa P 27ec1b26c2f64cc682ed7ae12e133a9f: Deleting WAL directory at /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-2-root/wals/2ecebe8dad3946bd956afacf695fabaa
I20240502 07:59:57.745290 951 catalog_manager.cc:4830] TS 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737): tablet 2ecebe8dad3946bd956afacf695fabaa (table test-workload [id=c4044eda63f741838cbc708a8054d3b9]) successfully deleted
I20240502 07:59:57.816459 2076 raft_consensus.cc:1050] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69: attempting to promote NON_VOTER 66df61031360493886f1170a809e079a to VOTER
I20240502 07:59:57.816934 2076 consensus_queue.cc:237] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 3.34, Last appended by leader: 32, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } }
I20240502 07:59:57.818393 1218 raft_consensus.cc:1261] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Refusing update from remote peer eb0a49d2a951470e85adbcd797128e69: Log matching property violated. Preceding OpId in replica: term: 3 index: 34. Preceding OpId from leader: term: 3 index: 35. (index mismatch)
I20240502 07:59:57.818403 1529 raft_consensus.cc:1261] T 69a9ebd0c390436b8ace08089f719358 P 66df61031360493886f1170a809e079a [term 3 LEARNER]: Refusing update from remote peer eb0a49d2a951470e85adbcd797128e69: Log matching property violated. Preceding OpId in replica: term: 3 index: 34. Preceding OpId from leader: term: 3 index: 35. (index mismatch)
I20240502 07:59:57.818761 2076 consensus_queue.cc:1035] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240502 07:59:57.818929 2077 consensus_queue.cc:1035] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240502 07:59:57.818956 1063 raft_consensus.cc:1261] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 FOLLOWER]: Refusing update from remote peer eb0a49d2a951470e85adbcd797128e69: Log matching property violated. Preceding OpId in replica: term: 3 index: 34. Preceding OpId from leader: term: 3 index: 35. (index mismatch)
I20240502 07:59:57.819337 2076 consensus_queue.cc:1035] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240502 07:59:57.820953 2034 raft_consensus.cc:2932] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [term 3 LEADER]: Committing config change with OpId 3.35: config changed from index 34 to 35, 66df61031360493886f1170a809e079a (127.29.106.133) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:57.821535 1063 raft_consensus.cc:2932] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 FOLLOWER]: Committing config change with OpId 3.35: config changed from index 34 to 35, 66df61031360493886f1170a809e079a (127.29.106.133) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:57.822714 1218 raft_consensus.cc:2932] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Committing config change with OpId 3.35: config changed from index 34 to 35, 66df61031360493886f1170a809e079a (127.29.106.133) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:57.822834 1529 raft_consensus.cc:2932] T 69a9ebd0c390436b8ace08089f719358 P 66df61031360493886f1170a809e079a [term 3 FOLLOWER]: Committing config change with OpId 3.35: config changed from index 34 to 35, 66df61031360493886f1170a809e079a (127.29.106.133) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:57.824757 961 catalog_manager.cc:5484] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 reported cstate change: config changed from index 34 to 35, 66df61031360493886f1170a809e079a (127.29.106.133) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "eb0a49d2a951470e85adbcd797128e69" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:57.827772 1604 consensus_queue.cc:237] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 3.35, Last appended by leader: 32, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } }
I20240502 07:59:57.828955 1529 raft_consensus.cc:1261] T 69a9ebd0c390436b8ace08089f719358 P 66df61031360493886f1170a809e079a [term 3 FOLLOWER]: Refusing update from remote peer eb0a49d2a951470e85adbcd797128e69: Log matching property violated. Preceding OpId in replica: term: 3 index: 35. Preceding OpId from leader: term: 3 index: 36. (index mismatch)
I20240502 07:59:57.829416 2076 consensus_queue.cc:1035] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20240502 07:59:57.829656 1063 raft_consensus.cc:1261] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 FOLLOWER]: Refusing update from remote peer eb0a49d2a951470e85adbcd797128e69: Log matching property violated. Preceding OpId in replica: term: 3 index: 35. Preceding OpId from leader: term: 3 index: 36. (index mismatch)
I20240502 07:59:57.829982 2076 consensus_queue.cc:1035] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20240502 07:59:57.831099 2079 raft_consensus.cc:2932] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [term 3 LEADER]: Committing config change with OpId 3.36: config changed from index 35 to 36, VOTER 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131) evicted. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:57.831527 1529 raft_consensus.cc:2932] T 69a9ebd0c390436b8ace08089f719358 P 66df61031360493886f1170a809e079a [term 3 FOLLOWER]: Committing config change with OpId 3.36: config changed from index 35 to 36, VOTER 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131) evicted. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:57.831772 1063 raft_consensus.cc:2932] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 FOLLOWER]: Committing config change with OpId 3.36: config changed from index 35 to 36, VOTER 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131) evicted. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:57.832795 1012 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 69a9ebd0c390436b8ace08089f719358 with cas_config_opid_index 35: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:57.834184 962 catalog_manager.cc:5484] T 69a9ebd0c390436b8ace08089f719358 P 66df61031360493886f1170a809e079a reported cstate change: config changed from index 35 to 36, VOTER 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131) evicted. New cstate: current_term: 3 leader_uuid: "eb0a49d2a951470e85adbcd797128e69" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "77e648af9dbb46fc881340efe1d1e3c3" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 34761 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:57.836493 1207 tablet_service.cc:1514] Processing DeleteTablet for tablet 69a9ebd0c390436b8ace08089f719358 with delete_type TABLET_DATA_TOMBSTONED (TS 27ec1b26c2f64cc682ed7ae12e133a9f not found in new config with opid_index 36) from {username='slave'} at 127.0.0.1:52036
I20240502 07:59:57.836678 2075 tablet_replica.cc:330] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f: stopping tablet replica
I20240502 07:59:57.836804 2075 raft_consensus.cc:2229] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:57.836953 2075 raft_consensus.cc:2258] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:57.837531 2075 ts_tablet_manager.cc:1875] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:57.839691 2075 ts_tablet_manager.cc:1888] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.35
I20240502 07:59:57.839777 2075 log.cc:1196] T 69a9ebd0c390436b8ace08089f719358 P 27ec1b26c2f64cc682ed7ae12e133a9f: Deleting WAL directory at /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-2-root/wals/69a9ebd0c390436b8ace08089f719358
I20240502 07:59:57.840302 951 catalog_manager.cc:4830] TS 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737): tablet 69a9ebd0c390436b8ace08089f719358 (table test-workload [id=c4044eda63f741838cbc708a8054d3b9]) successfully deleted
I20240502 07:59:57.874533 2049 raft_consensus.cc:1050] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07: attempting to promote NON_VOTER 66df61031360493886f1170a809e079a to VOTER
I20240502 07:59:57.875015 2049 consensus_queue.cc:237] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 31, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } }
I20240502 07:59:57.876484 1529 raft_consensus.cc:1261] T 2c1b6523d6724a02acc9ae19f3f0255e P 66df61031360493886f1170a809e079a [term 2 LEARNER]: Refusing update from remote peer 9d994926b3d54e6395a0d60df8a1db07: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240502 07:59:57.876490 1143 raft_consensus.cc:1261] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Refusing update from remote peer 9d994926b3d54e6395a0d60df8a1db07: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240502 07:59:57.876490 1218 raft_consensus.cc:1261] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Refusing update from remote peer 9d994926b3d54e6395a0d60df8a1db07: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240502 07:59:57.876899 2084 consensus_queue.cc:1035] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 07:59:57.877066 2049 consensus_queue.cc:1035] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 07:59:57.877249 2085 consensus_queue.cc:1035] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 07:59:57.879518 2085 raft_consensus.cc:2932] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [term 2 LEADER]: Committing config change with OpId 2.34: config changed from index 33 to 34, 66df61031360493886f1170a809e079a (127.29.106.133) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:57.879894 1218 raft_consensus.cc:2932] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, 66df61031360493886f1170a809e079a (127.29.106.133) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:57.881940 1529 raft_consensus.cc:2932] T 2c1b6523d6724a02acc9ae19f3f0255e P 66df61031360493886f1170a809e079a [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, 66df61031360493886f1170a809e079a (127.29.106.133) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:57.882377 955 catalog_manager.cc:5484] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 reported cstate change: config changed from index 33 to 34, 66df61031360493886f1170a809e079a (127.29.106.133) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "9d994926b3d54e6395a0d60df8a1db07" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:57.883733 1143 raft_consensus.cc:2932] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, 66df61031360493886f1170a809e079a (127.29.106.133) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "b1e9945160714b16b2db91f90c5ccc70" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:57.886137 1454 consensus_queue.cc:237] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 2.34, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } }
I20240502 07:59:57.887701 1529 raft_consensus.cc:1261] T 2c1b6523d6724a02acc9ae19f3f0255e P 66df61031360493886f1170a809e079a [term 2 FOLLOWER]: Refusing update from remote peer 9d994926b3d54e6395a0d60df8a1db07: Log matching property violated. Preceding OpId in replica: term: 2 index: 34. Preceding OpId from leader: term: 2 index: 35. (index mismatch)
I20240502 07:59:57.887964 1218 raft_consensus.cc:1261] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Refusing update from remote peer 9d994926b3d54e6395a0d60df8a1db07: Log matching property violated. Preceding OpId in replica: term: 2 index: 34. Preceding OpId from leader: term: 2 index: 35. (index mismatch)
I20240502 07:59:57.888012 2084 consensus_queue.cc:1035] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240502 07:59:57.888334 2002 consensus_queue.cc:1035] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240502 07:59:57.889503 2084 raft_consensus.cc:2932] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [term 2 LEADER]: Committing config change with OpId 2.35: config changed from index 34 to 35, VOTER b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:57.889916 1529 raft_consensus.cc:2932] T 2c1b6523d6724a02acc9ae19f3f0255e P 66df61031360493886f1170a809e079a [term 2 FOLLOWER]: Committing config change with OpId 2.35: config changed from index 34 to 35, VOTER b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:57.891644 1008 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2c1b6523d6724a02acc9ae19f3f0255e with cas_config_opid_index 34: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:57.892342 960 catalog_manager.cc:5484] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 reported cstate change: config changed from index 34 to 35, VOTER b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130) evicted. New cstate: current_term: 2 leader_uuid: "9d994926b3d54e6395a0d60df8a1db07" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:57.893790 1218 raft_consensus.cc:2932] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Committing config change with OpId 2.35: config changed from index 34 to 35, VOTER b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } } peers { permanent_uuid: "66df61031360493886f1170a809e079a" member_type: VOTER last_known_addr { host: "127.29.106.133" port: 38717 } attrs { promote: false } } }
I20240502 07:59:57.895478 1127 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c1b6523d6724a02acc9ae19f3f0255e with delete_type TABLET_DATA_TOMBSTONED (TS b1e9945160714b16b2db91f90c5ccc70 not found in new config with opid_index 35) from {username='slave'} at 127.0.0.1:47876
I20240502 07:59:57.895707 2053 tablet_replica.cc:330] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70: stopping tablet replica
I20240502 07:59:57.895845 2053 raft_consensus.cc:2229] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:57.895994 2053 raft_consensus.cc:2258] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:57.896903 2053 ts_tablet_manager.cc:1875] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:57.898530 2053 ts_tablet_manager.cc:1888] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.34
I20240502 07:59:57.898597 2053 log.cc:1196] T 2c1b6523d6724a02acc9ae19f3f0255e P b1e9945160714b16b2db91f90c5ccc70: Deleting WAL directory at /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714636688849691-30122-0/minicluster-data/ts-1-root/wals/2c1b6523d6724a02acc9ae19f3f0255e
I20240502 07:59:57.899068 948 catalog_manager.cc:4830] TS b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203): tablet 2c1b6523d6724a02acc9ae19f3f0255e (table test-workload [id=c4044eda63f741838cbc708a8054d3b9]) successfully deleted
I20240502 07:59:57.948729 1018 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:59:57.950631 1218 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6edf17550e3f4a0b945477edc6de6af1"
dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
mode: GRACEFUL
new_leader_uuid: "eb0a49d2a951470e85adbcd797128e69"
from {username='slave'} at 127.0.0.1:40314
I20240502 07:59:57.950819 1218 raft_consensus.cc:595] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 4 LEADER]: Received request to transfer leadership to TS eb0a49d2a951470e85adbcd797128e69
I20240502 07:59:57.951213 1018 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240502 07:59:57.951347 1018 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 07:59:57.970326 2035 raft_consensus.cc:979] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f: : Instructing follower eb0a49d2a951470e85adbcd797128e69 to start an election
I20240502 07:59:57.970438 2035 raft_consensus.cc:1067] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 4 LEADER]: Signalling peer eb0a49d2a951470e85adbcd797128e69 to start an election
I20240502 07:59:57.970927 1605 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6edf17550e3f4a0b945477edc6de6af1"
dest_uuid: "eb0a49d2a951470e85adbcd797128e69"
from {username='slave'} at 127.0.0.1:35580
I20240502 07:59:57.971052 1605 raft_consensus.cc:482] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:57.971093 1605 raft_consensus.cc:3037] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [term 4 FOLLOWER]: Advancing to term 5
I20240502 07:59:57.971969 1605 raft_consensus.cc:504] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } }
I20240502 07:59:57.972393 1605 leader_election.cc:288] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [CANDIDATE]: Term 5 election: Requested vote from peers 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737), 9d994926b3d54e6395a0d60df8a1db07 (127.29.106.132:44697)
I20240502 07:59:57.972688 1218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6edf17550e3f4a0b945477edc6de6af1" candidate_uuid: "eb0a49d2a951470e85adbcd797128e69" candidate_term: 5 candidate_status { last_received { term: 4 index: 37 } } ignore_live_leader: true dest_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f"
I20240502 07:59:57.972800 1218 raft_consensus.cc:3032] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 4 LEADER]: Stepping down as leader of term 4
I20240502 07:59:57.972848 1218 raft_consensus.cc:729] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 27ec1b26c2f64cc682ed7ae12e133a9f, State: Running, Role: LEADER
I20240502 07:59:57.973012 1218 consensus_queue.cc:260] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 4.37, Last appended by leader: 37, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } }
I20240502 07:59:57.973201 1218 raft_consensus.cc:3037] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 4 FOLLOWER]: Advancing to term 5
I20240502 07:59:57.974298 1218 raft_consensus.cc:2454] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb0a49d2a951470e85adbcd797128e69 in term 5.
I20240502 07:59:57.974670 1571 leader_election.cc:302] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ec1b26c2f64cc682ed7ae12e133a9f, eb0a49d2a951470e85adbcd797128e69; no voters:
I20240502 07:59:57.974874 2076 raft_consensus.cc:2781] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 07:59:57.974954 2076 raft_consensus.cc:686] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [term 5 LEADER]: Becoming Leader. State: Replica: eb0a49d2a951470e85adbcd797128e69, State: Running, Role: LEADER
I20240502 07:59:57.975112 2076 consensus_queue.cc:237] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 4.37, Last appended by leader: 37, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } }
I20240502 07:59:57.976763 962 catalog_manager.cc:5484] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 reported cstate change: term changed from 4 to 5, leader changed from 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131) to eb0a49d2a951470e85adbcd797128e69 (127.29.106.134). New cstate: current_term: 5 leader_uuid: "eb0a49d2a951470e85adbcd797128e69" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb0a49d2a951470e85adbcd797128e69" member_type: VOTER last_known_addr { host: "127.29.106.134" port: 34541 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:57.978744 1454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6edf17550e3f4a0b945477edc6de6af1" candidate_uuid: "eb0a49d2a951470e85adbcd797128e69" candidate_term: 5 candidate_status { last_received { term: 4 index: 37 } } ignore_live_leader: true dest_uuid: "9d994926b3d54e6395a0d60df8a1db07"
I20240502 07:59:57.978880 1454 raft_consensus.cc:3037] T 6edf17550e3f4a0b945477edc6de6af1 P 9d994926b3d54e6395a0d60df8a1db07 [term 4 FOLLOWER]: Advancing to term 5
I20240502 07:59:57.979655 1454 raft_consensus.cc:2454] T 6edf17550e3f4a0b945477edc6de6af1 P 9d994926b3d54e6395a0d60df8a1db07 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb0a49d2a951470e85adbcd797128e69 in term 5.
I20240502 07:59:58.411798 1454 raft_consensus.cc:1261] T 6edf17550e3f4a0b945477edc6de6af1 P 9d994926b3d54e6395a0d60df8a1db07 [term 5 FOLLOWER]: Refusing update from remote peer eb0a49d2a951470e85adbcd797128e69: Log matching property violated. Preceding OpId in replica: term: 4 index: 37. Preceding OpId from leader: term: 5 index: 38. (index mismatch)
I20240502 07:59:58.412233 2076 consensus_queue.cc:1035] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d994926b3d54e6395a0d60df8a1db07" member_type: VOTER last_known_addr { host: "127.29.106.132" port: 44697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20240502 07:59:58.414691 1218 raft_consensus.cc:1261] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 5 FOLLOWER]: Refusing update from remote peer eb0a49d2a951470e85adbcd797128e69: Log matching property violated. Preceding OpId in replica: term: 4 index: 37. Preceding OpId from leader: term: 5 index: 38. (index mismatch)
I20240502 07:59:58.415637 2077 consensus_queue.cc:1035] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ec1b26c2f64cc682ed7ae12e133a9f" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20240502 07:59:59.951740 1018 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 07:59:59.953101 1018 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 07:59:59.953224 1018 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:00:00.367547 30122 tablet_server.cc:178] TabletServer@127.29.106.129:0 shutting down...
I20240502 08:00:00.373529 30122 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:00:00.373812 30122 tablet_replica.cc:330] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3: stopping tablet replica
I20240502 08:00:00.373932 30122 raft_consensus.cc:2229] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:00:00.374147 30122 raft_consensus.cc:2258] T c7c1536618ae4e74aa79349d04201fb9 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.374624 30122 tablet_replica.cc:330] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3: stopping tablet replica
I20240502 08:00:00.374716 30122 raft_consensus.cc:2229] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:00.374791 30122 raft_consensus.cc:2258] T d0d982dd52d246348fb3984fd1c3a40f P 77e648af9dbb46fc881340efe1d1e3c3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.375097 30122 tablet_replica.cc:330] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3: stopping tablet replica
I20240502 08:00:00.375217 30122 raft_consensus.cc:2229] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:00.375371 30122 raft_consensus.cc:2258] T 69a9ebd0c390436b8ace08089f719358 P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.375648 30122 tablet_replica.cc:330] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3: stopping tablet replica
I20240502 08:00:00.375725 30122 raft_consensus.cc:2229] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:00:00.375862 30122 raft_consensus.cc:2258] T fda98cfe14f947969fb41e26c75bc384 P 77e648af9dbb46fc881340efe1d1e3c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.376135 30122 tablet_replica.cc:330] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3: stopping tablet replica
I20240502 08:00:00.376205 30122 raft_consensus.cc:2229] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:00.376318 30122 raft_consensus.cc:2258] T 58c5f52a420c467684b2cb50972e7c99 P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.376703 30122 tablet_replica.cc:330] T 5437f94679a841e3980053e69d624d76 P 77e648af9dbb46fc881340efe1d1e3c3: stopping tablet replica
I20240502 08:00:00.376778 30122 raft_consensus.cc:2229] T 5437f94679a841e3980053e69d624d76 P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:00.376883 30122 raft_consensus.cc:2258] T 5437f94679a841e3980053e69d624d76 P 77e648af9dbb46fc881340efe1d1e3c3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.393863 30122 tablet_server.cc:194] TabletServer@127.29.106.129:0 shutdown complete.
I20240502 08:00:00.397395 30122 tablet_server.cc:178] TabletServer@127.29.106.130:0 shutting down...
I20240502 08:00:00.403944 30122 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:00:00.404341 30122 tablet_replica.cc:330] T d0d982dd52d246348fb3984fd1c3a40f P b1e9945160714b16b2db91f90c5ccc70: stopping tablet replica
I20240502 08:00:00.404490 30122 raft_consensus.cc:2229] T d0d982dd52d246348fb3984fd1c3a40f P b1e9945160714b16b2db91f90c5ccc70 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:00.404681 30122 raft_consensus.cc:2258] T d0d982dd52d246348fb3984fd1c3a40f P b1e9945160714b16b2db91f90c5ccc70 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.405181 30122 tablet_replica.cc:330] T df4233b549b34b01b6f9662c7e7b3515 P b1e9945160714b16b2db91f90c5ccc70: stopping tablet replica
I20240502 08:00:00.405295 30122 raft_consensus.cc:2229] T df4233b549b34b01b6f9662c7e7b3515 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:00.405447 30122 raft_consensus.cc:2258] T df4233b549b34b01b6f9662c7e7b3515 P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.405755 30122 tablet_replica.cc:330] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70: stopping tablet replica
I20240502 08:00:00.405834 30122 raft_consensus.cc:2229] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [term 3 LEADER]: Raft consensus shutting down.
I20240502 08:00:00.406006 30122 raft_consensus.cc:2258] T 5437f94679a841e3980053e69d624d76 P b1e9945160714b16b2db91f90c5ccc70 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.406312 30122 tablet_replica.cc:330] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70: stopping tablet replica
I20240502 08:00:00.406388 30122 raft_consensus.cc:2229] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 3 LEADER]: Raft consensus shutting down.
I20240502 08:00:00.406515 30122 raft_consensus.cc:2258] T 2ecebe8dad3946bd956afacf695fabaa P b1e9945160714b16b2db91f90c5ccc70 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.406810 30122 tablet_replica.cc:330] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70: stopping tablet replica
I20240502 08:00:00.406883 30122 raft_consensus.cc:2229] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:00.406950 30122 raft_consensus.cc:2258] T cdf0ff2967c040a589f915172d3f446b P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.407243 30122 tablet_replica.cc:330] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70: stopping tablet replica
I20240502 08:00:00.407343 30122 raft_consensus.cc:2229] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:00.407514 30122 raft_consensus.cc:2258] T 682e26b333104f82863b92c545cfdafe P b1e9945160714b16b2db91f90c5ccc70 [term 2 FOLLOWER]: Raft consensus is shut down!
W20240502 08:00:00.408345 1494 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.106.129:34761: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240502 08:00:00.409231 1494 consensus_peers.cc:485] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a -> Peer 77e648af9dbb46fc881340efe1d1e3c3 (127.29.106.129:34761): Couldn't send request to peer 77e648af9dbb46fc881340efe1d1e3c3. Status: Network error: Client connection negotiation failed: client connection to 127.29.106.129:34761: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.422669 30122 tablet_server.cc:194] TabletServer@127.29.106.130:0 shutdown complete.
W20240502 08:00:00.426566 1417 consensus_peers.cc:485] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07 -> Peer b1e9945160714b16b2db91f90c5ccc70 (127.29.106.130:38203): Couldn't send request to peer b1e9945160714b16b2db91f90c5ccc70. Status: Network error: Client connection negotiation failed: client connection to 127.29.106.130:38203: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.427357 30122 tablet_server.cc:178] TabletServer@127.29.106.131:0 shutting down...
W20240502 08:00:00.432237 1420 consensus_peers.cc:485] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07 -> Peer 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737): Couldn't send request to peer 27ec1b26c2f64cc682ed7ae12e133a9f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.434273 30122 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:00:00.434572 30122 tablet_replica.cc:330] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f: stopping tablet replica
I20240502 08:00:00.434747 30122 raft_consensus.cc:2229] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:00.435009 30122 raft_consensus.cc:2258] T 2c1b6523d6724a02acc9ae19f3f0255e P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.435564 30122 tablet_replica.cc:330] T cdf0ff2967c040a589f915172d3f446b P 27ec1b26c2f64cc682ed7ae12e133a9f: stopping tablet replica
I20240502 08:00:00.435676 30122 raft_consensus.cc:2229] T cdf0ff2967c040a589f915172d3f446b P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:00.435853 30122 raft_consensus.cc:2258] T cdf0ff2967c040a589f915172d3f446b P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.436187 30122 tablet_replica.cc:330] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f: stopping tablet replica
I20240502 08:00:00.436311 30122 raft_consensus.cc:2229] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 LEADER]: Raft consensus shutting down.
I20240502 08:00:00.436499 30122 raft_consensus.cc:2258] T 682e26b333104f82863b92c545cfdafe P 27ec1b26c2f64cc682ed7ae12e133a9f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.436831 30122 tablet_replica.cc:330] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f: stopping tablet replica
I20240502 08:00:00.436901 30122 raft_consensus.cc:2229] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [term 4 LEADER]: Raft consensus shutting down.
I20240502 08:00:00.437067 30122 raft_consensus.cc:2258] T d0d982dd52d246348fb3984fd1c3a40f P 27ec1b26c2f64cc682ed7ae12e133a9f [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.437403 30122 tablet_replica.cc:330] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f: stopping tablet replica
I20240502 08:00:00.437476 30122 raft_consensus.cc:2229] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 5 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:00.437575 30122 raft_consensus.cc:2258] T 6edf17550e3f4a0b945477edc6de6af1 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.437894 30122 tablet_replica.cc:330] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f: stopping tablet replica
I20240502 08:00:00.437973 30122 raft_consensus.cc:2229] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:00.438107 30122 raft_consensus.cc:2258] T 58c5f52a420c467684b2cb50972e7c99 P 27ec1b26c2f64cc682ed7ae12e133a9f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.453159 30122 tablet_server.cc:194] TabletServer@127.29.106.131:0 shutdown complete.
I20240502 08:00:00.457919 30122 tablet_server.cc:178] TabletServer@127.29.106.132:0 shutting down...
I20240502 08:00:00.463423 30122 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:00:00.463706 30122 tablet_replica.cc:330] T 6edf17550e3f4a0b945477edc6de6af1 P 9d994926b3d54e6395a0d60df8a1db07: stopping tablet replica
I20240502 08:00:00.463886 30122 raft_consensus.cc:2229] T 6edf17550e3f4a0b945477edc6de6af1 P 9d994926b3d54e6395a0d60df8a1db07 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:00.464072 30122 raft_consensus.cc:2258] T 6edf17550e3f4a0b945477edc6de6af1 P 9d994926b3d54e6395a0d60df8a1db07 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.464695 30122 tablet_replica.cc:330] T 2ecebe8dad3946bd956afacf695fabaa P 9d994926b3d54e6395a0d60df8a1db07: stopping tablet replica
I20240502 08:00:00.464814 30122 raft_consensus.cc:2229] T 2ecebe8dad3946bd956afacf695fabaa P 9d994926b3d54e6395a0d60df8a1db07 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:00.464941 30122 raft_consensus.cc:2258] T 2ecebe8dad3946bd956afacf695fabaa P 9d994926b3d54e6395a0d60df8a1db07 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.465267 30122 tablet_replica.cc:330] T c7c1536618ae4e74aa79349d04201fb9 P 9d994926b3d54e6395a0d60df8a1db07: stopping tablet replica
I20240502 08:00:00.465368 30122 raft_consensus.cc:2229] T c7c1536618ae4e74aa79349d04201fb9 P 9d994926b3d54e6395a0d60df8a1db07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:00.465479 30122 raft_consensus.cc:2258] T c7c1536618ae4e74aa79349d04201fb9 P 9d994926b3d54e6395a0d60df8a1db07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.465814 30122 tablet_replica.cc:330] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07: stopping tablet replica
I20240502 08:00:00.465916 30122 raft_consensus.cc:2229] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07 [term 2 LEADER]: Raft consensus shutting down.
I20240502 08:00:00.466120 30122 raft_consensus.cc:2258] T cdf0ff2967c040a589f915172d3f446b P 9d994926b3d54e6395a0d60df8a1db07 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.466439 30122 tablet_replica.cc:330] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07: stopping tablet replica
I20240502 08:00:00.466540 30122 raft_consensus.cc:2229] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [term 2 LEADER]: Raft consensus shutting down.
I20240502 08:00:00.466696 30122 raft_consensus.cc:2258] T 2c1b6523d6724a02acc9ae19f3f0255e P 9d994926b3d54e6395a0d60df8a1db07 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.467017 30122 tablet_replica.cc:330] T fda98cfe14f947969fb41e26c75bc384 P 9d994926b3d54e6395a0d60df8a1db07: stopping tablet replica
I20240502 08:00:00.467113 30122 raft_consensus.cc:2229] T fda98cfe14f947969fb41e26c75bc384 P 9d994926b3d54e6395a0d60df8a1db07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:00.467248 30122 raft_consensus.cc:2258] T fda98cfe14f947969fb41e26c75bc384 P 9d994926b3d54e6395a0d60df8a1db07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.481518 30122 tablet_server.cc:194] TabletServer@127.29.106.132:0 shutdown complete.
I20240502 08:00:00.485204 30122 tablet_server.cc:178] TabletServer@127.29.106.133:0 shutting down...
I20240502 08:00:00.490008 30122 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:00:00.490268 30122 tablet_replica.cc:330] T 2c1b6523d6724a02acc9ae19f3f0255e P 66df61031360493886f1170a809e079a: stopping tablet replica
I20240502 08:00:00.490463 30122 raft_consensus.cc:2229] T 2c1b6523d6724a02acc9ae19f3f0255e P 66df61031360493886f1170a809e079a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:00.490656 30122 raft_consensus.cc:2258] T 2c1b6523d6724a02acc9ae19f3f0255e P 66df61031360493886f1170a809e079a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.491235 30122 tablet_replica.cc:330] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a: stopping tablet replica
I20240502 08:00:00.491354 30122 raft_consensus.cc:2229] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a [term 3 LEADER]: Raft consensus shutting down.
I20240502 08:00:00.491573 30122 raft_consensus.cc:2258] T 58c5f52a420c467684b2cb50972e7c99 P 66df61031360493886f1170a809e079a [term 3 FOLLOWER]: Raft consensus is shut down!
W20240502 08:00:00.492013 1571 consensus_peers.cc:485] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 -> Peer 27ec1b26c2f64cc682ed7ae12e133a9f (127.29.106.131:36737): Couldn't send request to peer 27ec1b26c2f64cc682ed7ae12e133a9f. Status: Network error: Client connection negotiation failed: client connection to 127.29.106.131:36737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.492017 30122 tablet_replica.cc:330] T c7c1536618ae4e74aa79349d04201fb9 P 66df61031360493886f1170a809e079a: stopping tablet replica
I20240502 08:00:00.492178 30122 raft_consensus.cc:2229] T c7c1536618ae4e74aa79349d04201fb9 P 66df61031360493886f1170a809e079a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:00.492401 30122 raft_consensus.cc:2258] T c7c1536618ae4e74aa79349d04201fb9 P 66df61031360493886f1170a809e079a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.492754 30122 tablet_replica.cc:330] T 69a9ebd0c390436b8ace08089f719358 P 66df61031360493886f1170a809e079a: stopping tablet replica
I20240502 08:00:00.492830 30122 raft_consensus.cc:2229] T 69a9ebd0c390436b8ace08089f719358 P 66df61031360493886f1170a809e079a [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:00.492921 30122 raft_consensus.cc:2258] T 69a9ebd0c390436b8ace08089f719358 P 66df61031360493886f1170a809e079a [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.493233 30122 tablet_replica.cc:330] T 2ecebe8dad3946bd956afacf695fabaa P 66df61031360493886f1170a809e079a: stopping tablet replica
I20240502 08:00:00.493307 30122 raft_consensus.cc:2229] T 2ecebe8dad3946bd956afacf695fabaa P 66df61031360493886f1170a809e079a [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:00.493448 30122 raft_consensus.cc:2258] T 2ecebe8dad3946bd956afacf695fabaa P 66df61031360493886f1170a809e079a [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.493762 30122 tablet_replica.cc:330] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a: stopping tablet replica
I20240502 08:00:00.493834 30122 raft_consensus.cc:2229] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [term 2 LEADER]: Raft consensus shutting down.
I20240502 08:00:00.494006 30122 raft_consensus.cc:2258] T df4233b549b34b01b6f9662c7e7b3515 P 66df61031360493886f1170a809e079a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.507942 30122 tablet_server.cc:194] TabletServer@127.29.106.133:0 shutdown complete.
I20240502 08:00:00.511008 30122 tablet_server.cc:178] TabletServer@127.29.106.134:0 shutting down...
I20240502 08:00:00.516635 30122 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:00:00.516897 30122 tablet_replica.cc:330] T 5437f94679a841e3980053e69d624d76 P eb0a49d2a951470e85adbcd797128e69: stopping tablet replica
I20240502 08:00:00.517046 30122 raft_consensus.cc:2229] T 5437f94679a841e3980053e69d624d76 P eb0a49d2a951470e85adbcd797128e69 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:00.517197 30122 raft_consensus.cc:2258] T 5437f94679a841e3980053e69d624d76 P eb0a49d2a951470e85adbcd797128e69 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.517647 30122 tablet_replica.cc:330] T fda98cfe14f947969fb41e26c75bc384 P eb0a49d2a951470e85adbcd797128e69: stopping tablet replica
I20240502 08:00:00.517740 30122 raft_consensus.cc:2229] T fda98cfe14f947969fb41e26c75bc384 P eb0a49d2a951470e85adbcd797128e69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:00.517830 30122 raft_consensus.cc:2258] T fda98cfe14f947969fb41e26c75bc384 P eb0a49d2a951470e85adbcd797128e69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.518129 30122 tablet_replica.cc:330] T df4233b549b34b01b6f9662c7e7b3515 P eb0a49d2a951470e85adbcd797128e69: stopping tablet replica
I20240502 08:00:00.518208 30122 raft_consensus.cc:2229] T df4233b549b34b01b6f9662c7e7b3515 P eb0a49d2a951470e85adbcd797128e69 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:00.518286 30122 raft_consensus.cc:2258] T df4233b549b34b01b6f9662c7e7b3515 P eb0a49d2a951470e85adbcd797128e69 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.518541 30122 tablet_replica.cc:330] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69: stopping tablet replica
I20240502 08:00:00.518615 30122 raft_consensus.cc:2229] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [term 3 LEADER]: Raft consensus shutting down.
I20240502 08:00:00.518783 30122 raft_consensus.cc:2258] T 69a9ebd0c390436b8ace08089f719358 P eb0a49d2a951470e85adbcd797128e69 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.519070 30122 tablet_replica.cc:330] T 682e26b333104f82863b92c545cfdafe P eb0a49d2a951470e85adbcd797128e69: stopping tablet replica
I20240502 08:00:00.519138 30122 raft_consensus.cc:2229] T 682e26b333104f82863b92c545cfdafe P eb0a49d2a951470e85adbcd797128e69 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:00.519227 30122 raft_consensus.cc:2258] T 682e26b333104f82863b92c545cfdafe P eb0a49d2a951470e85adbcd797128e69 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.519486 30122 tablet_replica.cc:330] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69: stopping tablet replica
I20240502 08:00:00.519554 30122 raft_consensus.cc:2229] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [term 5 LEADER]: Raft consensus shutting down.
I20240502 08:00:00.519706 30122 raft_consensus.cc:2258] T 6edf17550e3f4a0b945477edc6de6af1 P eb0a49d2a951470e85adbcd797128e69 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.533588 30122 tablet_server.cc:194] TabletServer@127.29.106.134:0 shutdown complete.
I20240502 08:00:00.537392 30122 master.cc:537] Master@127.29.106.190:35263 shutting down...
I20240502 08:00:00.542196 30122 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 004b7395c3984a0d9d1918160d8b4233 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:00:00.542402 30122 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 004b7395c3984a0d9d1918160d8b4233 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:00.542469 30122 tablet_replica.cc:330] T 00000000000000000000000000000000 P 004b7395c3984a0d9d1918160d8b4233: stopping tablet replica
I20240502 08:00:00.556430 30122 master.cc:552] Master@127.29.106.190:35263 shutdown complete.
[ OK ] AutoRebalancerTest.TestMaxMovesPerServer (18806 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingUnstableCluster
I20240502 08:00:00.573324 30122 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.106.190:45161
I20240502 08:00:00.573684 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 08:00:00.576586 30122 server_base.cc:1034] running on GCE node
W20240502 08:00:00.577186 2105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:00:00.577845 2103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:00:00.578783 2102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:00:00.579092 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:00:00.579145 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:00:00.579164 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636800579163 us; error 0 us; skew 500 ppm
I20240502 08:00:00.584491 30122 webserver.cc:460] Webserver started at http://127.29.106.190:43449/ using document root <none> and password file <none>
I20240502 08:00:00.584702 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 08:00:00.584762 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:00:00.584843 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:00:00.585309 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636688849691-30122-0/minicluster-data/master-0-root/instance:
uuid: "0e303048dc624d6b98ec709370642523"
format_stamp: "Formatted at 2024-05-02 08:00:00 on dist-test-slave-m6l2"
I20240502 08:00:00.586970 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 08:00:00.588075 2110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:00:00.588526 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 08:00:00.588622 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636688849691-30122-0/minicluster-data/master-0-root
uuid: "0e303048dc624d6b98ec709370642523"
format_stamp: "Formatted at 2024-05-02 08:00:00 on dist-test-slave-m6l2"
I20240502 08:00:00.588701 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636688849691-30122-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636688849691-30122-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636688849691-30122-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:00:00.597395 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:00:00.597852 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:00:00.602929 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.190:45161
I20240502 08:00:00.611918 2162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.190:45161 every 8 connection(s)
I20240502 08:00:00.612488 2163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:00.614602 2163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e303048dc624d6b98ec709370642523: Bootstrap starting.
I20240502 08:00:00.615449 2163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e303048dc624d6b98ec709370642523: Neither blocks nor log segments found. Creating new log.
I20240502 08:00:00.616865 2163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e303048dc624d6b98ec709370642523: No bootstrap required, opened a new log
I20240502 08:00:00.617287 2163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e303048dc624d6b98ec709370642523 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e303048dc624d6b98ec709370642523" member_type: VOTER }
I20240502 08:00:00.617385 2163 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e303048dc624d6b98ec709370642523 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:00:00.617421 2163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e303048dc624d6b98ec709370642523 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e303048dc624d6b98ec709370642523, State: Initialized, Role: FOLLOWER
I20240502 08:00:00.617556 2163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e303048dc624d6b98ec709370642523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e303048dc624d6b98ec709370642523" member_type: VOTER }
I20240502 08:00:00.617622 2163 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e303048dc624d6b98ec709370642523 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:00:00.617659 2163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e303048dc624d6b98ec709370642523 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:00:00.617720 2163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e303048dc624d6b98ec709370642523 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:00:00.618474 2163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e303048dc624d6b98ec709370642523 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e303048dc624d6b98ec709370642523" member_type: VOTER }
I20240502 08:00:00.618599 2163 leader_election.cc:302] T 00000000000000000000000000000000 P 0e303048dc624d6b98ec709370642523 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e303048dc624d6b98ec709370642523; no voters:
I20240502 08:00:00.618816 2163 leader_election.cc:288] T 00000000000000000000000000000000 P 0e303048dc624d6b98ec709370642523 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:00:00.618986 2166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e303048dc624d6b98ec709370642523 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:00:00.619271 2166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e303048dc624d6b98ec709370642523 [term 1 LEADER]: Becoming Leader. State: Replica: 0e303048dc624d6b98ec709370642523, State: Running, Role: LEADER
I20240502 08:00:00.619299 2163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e303048dc624d6b98ec709370642523 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:00:00.619444 2166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e303048dc624d6b98ec709370642523 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e303048dc624d6b98ec709370642523" member_type: VOTER }
I20240502 08:00:00.619964 2167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e303048dc624d6b98ec709370642523 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e303048dc624d6b98ec709370642523" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e303048dc624d6b98ec709370642523" member_type: VOTER } }
I20240502 08:00:00.620081 2167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e303048dc624d6b98ec709370642523 [sys.catalog]: This master's current role is: LEADER
I20240502 08:00:00.619982 2168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e303048dc624d6b98ec709370642523 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e303048dc624d6b98ec709370642523. Latest consensus state: current_term: 1 leader_uuid: "0e303048dc624d6b98ec709370642523" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e303048dc624d6b98ec709370642523" member_type: VOTER } }
I20240502 08:00:00.620206 2168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e303048dc624d6b98ec709370642523 [sys.catalog]: This master's current role is: LEADER
I20240502 08:00:00.620862 2174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:00:00.621678 2174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:00:00.621901 30122 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 08:00:00.623872 2174 catalog_manager.cc:1347] Generated new cluster ID: d3f7a0ac2ca6470ca0e011e33d0ef46c
I20240502 08:00:00.623941 2174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:00:00.630416 2174 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:00:00.631052 2174 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:00:00.638082 2174 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 0e303048dc624d6b98ec709370642523: Generated new TSK 0
I20240502 08:00:00.638278 2174 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:00:00.654623 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:00:00.657984 2187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:00:00.658571 2184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:00:00.659260 2185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:00:00.659864 30122 server_base.cc:1034] running on GCE node
I20240502 08:00:00.660037 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:00:00.660075 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:00:00.660092 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636800660092 us; error 0 us; skew 500 ppm
I20240502 08:00:00.661353 30122 webserver.cc:460] Webserver started at http://127.29.106.129:40613/ using document root <none> and password file <none>
I20240502 08:00:00.661501 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 08:00:00.661550 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:00:00.661621 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:00:00.662022 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636688849691-30122-0/minicluster-data/ts-0-root/instance:
uuid: "3e4383ae832b477c8129bbb4fe162a0e"
format_stamp: "Formatted at 2024-05-02 08:00:00 on dist-test-slave-m6l2"
I20240502 08:00:00.663623 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 08:00:00.664906 2192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:00:00.665268 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 08:00:00.665472 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636688849691-30122-0/minicluster-data/ts-0-root
uuid: "3e4383ae832b477c8129bbb4fe162a0e"
format_stamp: "Formatted at 2024-05-02 08:00:00 on dist-test-slave-m6l2"
I20240502 08:00:00.665580 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636688849691-30122-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636688849691-30122-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636688849691-30122-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:00:00.680371 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:00:00.680780 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:00:00.681082 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:00:00.681557 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:00:00.681607 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:00:00.681656 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:00:00.681694 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:00:00.685894 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.129:33195
I20240502 08:00:00.686744 2255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.129:33195 every 8 connection(s)
I20240502 08:00:00.687158 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:00:00.690135 2261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:00:00.690843 2263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:00:00.691439 2260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:00:00.691995 2256 heartbeater.cc:347] Connected to a master server at 127.29.106.190:45161
I20240502 08:00:00.692126 2256 heartbeater.cc:464] Registering TS with master...
I20240502 08:00:00.692148 30122 server_base.cc:1034] running on GCE node
I20240502 08:00:00.692308 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:00:00.692353 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:00:00.692368 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636800692367 us; error 0 us; skew 500 ppm
I20240502 08:00:00.692435 2256 heartbeater.cc:511] Master 127.29.106.190:45161 requested a full tablet report, sending...
I20240502 08:00:00.693076 2127 ts_manager.cc:194] Registered new tserver with Master: 3e4383ae832b477c8129bbb4fe162a0e (127.29.106.129:33195)
I20240502 08:00:00.693408 30122 webserver.cc:460] Webserver started at http://127.29.106.130:43701/ using document root <none> and password file <none>
I20240502 08:00:00.693570 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 08:00:00.693634 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:00:00.693707 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:00:00.693935 2127 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42882
I20240502 08:00:00.694129 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636688849691-30122-0/minicluster-data/ts-1-root/instance:
uuid: "2eea07a6840f4ca9997cb720a558d378"
format_stamp: "Formatted at 2024-05-02 08:00:00 on dist-test-slave-m6l2"
I20240502 08:00:00.695569 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 08:00:00.696496 2268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:00:00.696764 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 08:00:00.696835 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636688849691-30122-0/minicluster-data/ts-1-root
uuid: "2eea07a6840f4ca9997cb720a558d378"
format_stamp: "Formatted at 2024-05-02 08:00:00 on dist-test-slave-m6l2"
I20240502 08:00:00.696910 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636688849691-30122-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636688849691-30122-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636688849691-30122-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:00:00.716879 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:00:00.717280 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:00:00.717566 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:00:00.718037 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:00:00.718078 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:00:00.718125 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:00:00.718158 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:00:00.722381 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.130:45293
I20240502 08:00:00.722427 2331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.130:45293 every 8 connection(s)
I20240502 08:00:00.723172 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:00:00.725999 2335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:00:00.726712 2338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:00:00.727355 2332 heartbeater.cc:347] Connected to a master server at 127.29.106.190:45161
I20240502 08:00:00.727459 2332 heartbeater.cc:464] Registering TS with master...
I20240502 08:00:00.727712 30122 server_base.cc:1034] running on GCE node
I20240502 08:00:00.727716 2332 heartbeater.cc:511] Master 127.29.106.190:45161 requested a full tablet report, sending...
W20240502 08:00:00.728420 2336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:00:00.728498 2127 ts_manager.cc:194] Registered new tserver with Master: 2eea07a6840f4ca9997cb720a558d378 (127.29.106.130:45293)
I20240502 08:00:00.728653 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:00:00.728704 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:00:00.728720 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636800728720 us; error 0 us; skew 500 ppm
I20240502 08:00:00.729108 2127 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42898
I20240502 08:00:00.729818 30122 webserver.cc:460] Webserver started at http://127.29.106.131:45199/ using document root <none> and password file <none>
I20240502 08:00:00.729976 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 08:00:00.730027 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:00:00.730082 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:00:00.730443 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636688849691-30122-0/minicluster-data/ts-2-root/instance:
uuid: "0994655074c74efea4418291148fb7d7"
format_stamp: "Formatted at 2024-05-02 08:00:00 on dist-test-slave-m6l2"
I20240502 08:00:00.731931 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 08:00:00.732986 2343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:00:00.733260 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 08:00:00.733395 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636688849691-30122-0/minicluster-data/ts-2-root
uuid: "0994655074c74efea4418291148fb7d7"
format_stamp: "Formatted at 2024-05-02 08:00:00 on dist-test-slave-m6l2"
I20240502 08:00:00.733491 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636688849691-30122-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636688849691-30122-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636688849691-30122-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:00:00.745859 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:00:00.746299 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:00:00.746596 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:00:00.747095 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:00:00.747136 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:00:00.747186 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:00:00.747220 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:00:00.751545 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.131:33267
I20240502 08:00:00.751571 2406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.131:33267 every 8 connection(s)
I20240502 08:00:00.756520 2407 heartbeater.cc:347] Connected to a master server at 127.29.106.190:45161
I20240502 08:00:00.756650 2407 heartbeater.cc:464] Registering TS with master...
I20240502 08:00:00.756901 2407 heartbeater.cc:511] Master 127.29.106.190:45161 requested a full tablet report, sending...
I20240502 08:00:00.757619 2127 ts_manager.cc:194] Registered new tserver with Master: 0994655074c74efea4418291148fb7d7 (127.29.106.131:33267)
I20240502 08:00:00.757668 30122 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005693787s
I20240502 08:00:00.758313 2127 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42902
I20240502 08:00:01.694864 2256 heartbeater.cc:503] Master 127.29.106.190:45161 was elected leader, sending a full tablet report...
I20240502 08:00:01.730144 2332 heartbeater.cc:503] Master 127.29.106.190:45161 was elected leader, sending a full tablet report...
I20240502 08:00:01.759354 2407 heartbeater.cc:503] Master 127.29.106.190:45161 was elected leader, sending a full tablet report...
I20240502 08:00:01.784372 30122 test_util.cc:268] Using random seed: -42124754
I20240502 08:00:01.795351 2127 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42918:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:00:01.796180 2127 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:00:01.810667 2219 tablet_service.cc:1467] Processing CreateTablet for tablet e2ec882e67844f4faebdedf5e64973c9 (DEFAULT_TABLE table=test-workload [id=1facbd5fb7d44369a46308d914c7bca7]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240502 08:00:01.810667 2220 tablet_service.cc:1467] Processing CreateTablet for tablet 1a2593362f42463f80ea79b73a99e592 (DEFAULT_TABLE table=test-workload [id=1facbd5fb7d44369a46308d914c7bca7]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240502 08:00:01.810839 2371 tablet_service.cc:1467] Processing CreateTablet for tablet 1a2593362f42463f80ea79b73a99e592 (DEFAULT_TABLE table=test-workload [id=1facbd5fb7d44369a46308d914c7bca7]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240502 08:00:01.811224 2220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a2593362f42463f80ea79b73a99e592. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:01.811264 2371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a2593362f42463f80ea79b73a99e592. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:01.811677 2295 tablet_service.cc:1467] Processing CreateTablet for tablet e2ec882e67844f4faebdedf5e64973c9 (DEFAULT_TABLE table=test-workload [id=1facbd5fb7d44369a46308d914c7bca7]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240502 08:00:01.811712 2294 tablet_service.cc:1467] Processing CreateTablet for tablet de216561c2c04e3bbb8ea066078e9a40 (DEFAULT_TABLE table=test-workload [id=1facbd5fb7d44369a46308d914c7bca7]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240502 08:00:01.811975 2295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2ec882e67844f4faebdedf5e64973c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:01.812424 2294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de216561c2c04e3bbb8ea066078e9a40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:01.815610 2218 tablet_service.cc:1467] Processing CreateTablet for tablet de216561c2c04e3bbb8ea066078e9a40 (DEFAULT_TABLE table=test-workload [id=1facbd5fb7d44369a46308d914c7bca7]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240502 08:00:01.816041 2296 tablet_service.cc:1467] Processing CreateTablet for tablet 1a2593362f42463f80ea79b73a99e592 (DEFAULT_TABLE table=test-workload [id=1facbd5fb7d44369a46308d914c7bca7]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240502 08:00:01.816344 2218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de216561c2c04e3bbb8ea066078e9a40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:01.816416 2296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a2593362f42463f80ea79b73a99e592. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:01.816476 2429 tablet_bootstrap.cc:492] T 1a2593362f42463f80ea79b73a99e592 P 0994655074c74efea4418291148fb7d7: Bootstrap starting.
I20240502 08:00:01.817485 2429 tablet_bootstrap.cc:654] T 1a2593362f42463f80ea79b73a99e592 P 0994655074c74efea4418291148fb7d7: Neither blocks nor log segments found. Creating new log.
I20240502 08:00:01.818382 2428 tablet_bootstrap.cc:492] T 1a2593362f42463f80ea79b73a99e592 P 3e4383ae832b477c8129bbb4fe162a0e: Bootstrap starting.
I20240502 08:00:01.819480 2428 tablet_bootstrap.cc:654] T 1a2593362f42463f80ea79b73a99e592 P 3e4383ae832b477c8129bbb4fe162a0e: Neither blocks nor log segments found. Creating new log.
I20240502 08:00:01.819705 2370 tablet_service.cc:1467] Processing CreateTablet for tablet e2ec882e67844f4faebdedf5e64973c9 (DEFAULT_TABLE table=test-workload [id=1facbd5fb7d44369a46308d914c7bca7]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240502 08:00:01.819829 2219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2ec882e67844f4faebdedf5e64973c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:01.820042 2370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2ec882e67844f4faebdedf5e64973c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:01.820309 2427 tablet_bootstrap.cc:492] T e2ec882e67844f4faebdedf5e64973c9 P 2eea07a6840f4ca9997cb720a558d378: Bootstrap starting.
I20240502 08:00:01.821427 2429 tablet_bootstrap.cc:492] T 1a2593362f42463f80ea79b73a99e592 P 0994655074c74efea4418291148fb7d7: No bootstrap required, opened a new log
I20240502 08:00:01.821558 2429 ts_tablet_manager.cc:1367] T 1a2593362f42463f80ea79b73a99e592 P 0994655074c74efea4418291148fb7d7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:00:01.822081 2429 raft_consensus.cc:348] T 1a2593362f42463f80ea79b73a99e592 P 0994655074c74efea4418291148fb7d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } } peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } }
I20240502 08:00:01.822204 2429 raft_consensus.cc:374] T 1a2593362f42463f80ea79b73a99e592 P 0994655074c74efea4418291148fb7d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:00:01.822237 2429 raft_consensus.cc:729] T 1a2593362f42463f80ea79b73a99e592 P 0994655074c74efea4418291148fb7d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0994655074c74efea4418291148fb7d7, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.822383 2429 consensus_queue.cc:260] T 1a2593362f42463f80ea79b73a99e592 P 0994655074c74efea4418291148fb7d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } } peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } }
I20240502 08:00:01.822842 2429 ts_tablet_manager.cc:1398] T 1a2593362f42463f80ea79b73a99e592 P 0994655074c74efea4418291148fb7d7: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 08:00:01.823102 2429 tablet_bootstrap.cc:492] T e2ec882e67844f4faebdedf5e64973c9 P 0994655074c74efea4418291148fb7d7: Bootstrap starting.
I20240502 08:00:01.823271 2369 tablet_service.cc:1467] Processing CreateTablet for tablet de216561c2c04e3bbb8ea066078e9a40 (DEFAULT_TABLE table=test-workload [id=1facbd5fb7d44369a46308d914c7bca7]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240502 08:00:01.823549 2369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de216561c2c04e3bbb8ea066078e9a40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:01.824079 2429 tablet_bootstrap.cc:654] T e2ec882e67844f4faebdedf5e64973c9 P 0994655074c74efea4418291148fb7d7: Neither blocks nor log segments found. Creating new log.
I20240502 08:00:01.825160 2427 tablet_bootstrap.cc:654] T e2ec882e67844f4faebdedf5e64973c9 P 2eea07a6840f4ca9997cb720a558d378: Neither blocks nor log segments found. Creating new log.
I20240502 08:00:01.825733 2429 tablet_bootstrap.cc:492] T e2ec882e67844f4faebdedf5e64973c9 P 0994655074c74efea4418291148fb7d7: No bootstrap required, opened a new log
I20240502 08:00:01.825906 2429 ts_tablet_manager.cc:1367] T e2ec882e67844f4faebdedf5e64973c9 P 0994655074c74efea4418291148fb7d7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:00:01.826388 2429 raft_consensus.cc:348] T e2ec882e67844f4faebdedf5e64973c9 P 0994655074c74efea4418291148fb7d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } } peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } }
I20240502 08:00:01.826493 2429 raft_consensus.cc:374] T e2ec882e67844f4faebdedf5e64973c9 P 0994655074c74efea4418291148fb7d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:00:01.826527 2429 raft_consensus.cc:729] T e2ec882e67844f4faebdedf5e64973c9 P 0994655074c74efea4418291148fb7d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0994655074c74efea4418291148fb7d7, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.826670 2429 consensus_queue.cc:260] T e2ec882e67844f4faebdedf5e64973c9 P 0994655074c74efea4418291148fb7d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } } peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } }
I20240502 08:00:01.826926 2429 ts_tablet_manager.cc:1398] T e2ec882e67844f4faebdedf5e64973c9 P 0994655074c74efea4418291148fb7d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:00:01.827101 2429 tablet_bootstrap.cc:492] T de216561c2c04e3bbb8ea066078e9a40 P 0994655074c74efea4418291148fb7d7: Bootstrap starting.
I20240502 08:00:01.827877 2427 tablet_bootstrap.cc:492] T e2ec882e67844f4faebdedf5e64973c9 P 2eea07a6840f4ca9997cb720a558d378: No bootstrap required, opened a new log
I20240502 08:00:01.828058 2427 ts_tablet_manager.cc:1367] T e2ec882e67844f4faebdedf5e64973c9 P 2eea07a6840f4ca9997cb720a558d378: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 08:00:01.828065 2429 tablet_bootstrap.cc:654] T de216561c2c04e3bbb8ea066078e9a40 P 0994655074c74efea4418291148fb7d7: Neither blocks nor log segments found. Creating new log.
I20240502 08:00:01.828722 2428 tablet_bootstrap.cc:492] T 1a2593362f42463f80ea79b73a99e592 P 3e4383ae832b477c8129bbb4fe162a0e: No bootstrap required, opened a new log
I20240502 08:00:01.828821 2428 ts_tablet_manager.cc:1367] T 1a2593362f42463f80ea79b73a99e592 P 3e4383ae832b477c8129bbb4fe162a0e: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240502 08:00:01.828907 2427 raft_consensus.cc:348] T e2ec882e67844f4faebdedf5e64973c9 P 2eea07a6840f4ca9997cb720a558d378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } } peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } }
I20240502 08:00:01.829053 2427 raft_consensus.cc:374] T e2ec882e67844f4faebdedf5e64973c9 P 2eea07a6840f4ca9997cb720a558d378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:00:01.829095 2427 raft_consensus.cc:729] T e2ec882e67844f4faebdedf5e64973c9 P 2eea07a6840f4ca9997cb720a558d378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eea07a6840f4ca9997cb720a558d378, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.829238 2427 consensus_queue.cc:260] T e2ec882e67844f4faebdedf5e64973c9 P 2eea07a6840f4ca9997cb720a558d378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } } peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } }
I20240502 08:00:01.829227 2428 raft_consensus.cc:348] T 1a2593362f42463f80ea79b73a99e592 P 3e4383ae832b477c8129bbb4fe162a0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } } peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } }
I20240502 08:00:01.829358 2428 raft_consensus.cc:374] T 1a2593362f42463f80ea79b73a99e592 P 3e4383ae832b477c8129bbb4fe162a0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:00:01.829401 2428 raft_consensus.cc:729] T 1a2593362f42463f80ea79b73a99e592 P 3e4383ae832b477c8129bbb4fe162a0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e4383ae832b477c8129bbb4fe162a0e, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.829507 2429 tablet_bootstrap.cc:492] T de216561c2c04e3bbb8ea066078e9a40 P 0994655074c74efea4418291148fb7d7: No bootstrap required, opened a new log
I20240502 08:00:01.829576 2429 ts_tablet_manager.cc:1367] T de216561c2c04e3bbb8ea066078e9a40 P 0994655074c74efea4418291148fb7d7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:00:01.829538 2428 consensus_queue.cc:260] T 1a2593362f42463f80ea79b73a99e592 P 3e4383ae832b477c8129bbb4fe162a0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } } peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } }
I20240502 08:00:01.829814 2427 ts_tablet_manager.cc:1398] T e2ec882e67844f4faebdedf5e64973c9 P 2eea07a6840f4ca9997cb720a558d378: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:00:01.829895 2428 ts_tablet_manager.cc:1398] T 1a2593362f42463f80ea79b73a99e592 P 3e4383ae832b477c8129bbb4fe162a0e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:00:01.829984 2427 tablet_bootstrap.cc:492] T de216561c2c04e3bbb8ea066078e9a40 P 2eea07a6840f4ca9997cb720a558d378: Bootstrap starting.
I20240502 08:00:01.830088 2428 tablet_bootstrap.cc:492] T de216561c2c04e3bbb8ea066078e9a40 P 3e4383ae832b477c8129bbb4fe162a0e: Bootstrap starting.
I20240502 08:00:01.831086 2428 tablet_bootstrap.cc:654] T de216561c2c04e3bbb8ea066078e9a40 P 3e4383ae832b477c8129bbb4fe162a0e: Neither blocks nor log segments found. Creating new log.
I20240502 08:00:01.831197 2427 tablet_bootstrap.cc:654] T de216561c2c04e3bbb8ea066078e9a40 P 2eea07a6840f4ca9997cb720a558d378: Neither blocks nor log segments found. Creating new log.
I20240502 08:00:01.831557 2436 raft_consensus.cc:482] T e2ec882e67844f4faebdedf5e64973c9 P 2eea07a6840f4ca9997cb720a558d378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:00:01.831650 2436 raft_consensus.cc:504] T e2ec882e67844f4faebdedf5e64973c9 P 2eea07a6840f4ca9997cb720a558d378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } } peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } }
I20240502 08:00:01.832158 2429 raft_consensus.cc:348] T de216561c2c04e3bbb8ea066078e9a40 P 0994655074c74efea4418291148fb7d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } } peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } }
I20240502 08:00:01.832340 2429 raft_consensus.cc:374] T de216561c2c04e3bbb8ea066078e9a40 P 0994655074c74efea4418291148fb7d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:00:01.832405 2429 raft_consensus.cc:729] T de216561c2c04e3bbb8ea066078e9a40 P 0994655074c74efea4418291148fb7d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0994655074c74efea4418291148fb7d7, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.832556 2429 consensus_queue.cc:260] T de216561c2c04e3bbb8ea066078e9a40 P 0994655074c74efea4418291148fb7d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } } peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } }
I20240502 08:00:01.832700 2428 tablet_bootstrap.cc:492] T de216561c2c04e3bbb8ea066078e9a40 P 3e4383ae832b477c8129bbb4fe162a0e: No bootstrap required, opened a new log
I20240502 08:00:01.832746 2427 tablet_bootstrap.cc:492] T de216561c2c04e3bbb8ea066078e9a40 P 2eea07a6840f4ca9997cb720a558d378: No bootstrap required, opened a new log
I20240502 08:00:01.832784 2428 ts_tablet_manager.cc:1367] T de216561c2c04e3bbb8ea066078e9a40 P 3e4383ae832b477c8129bbb4fe162a0e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:00:01.832831 2427 ts_tablet_manager.cc:1367] T de216561c2c04e3bbb8ea066078e9a40 P 2eea07a6840f4ca9997cb720a558d378: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:00:01.833226 2428 raft_consensus.cc:348] T de216561c2c04e3bbb8ea066078e9a40 P 3e4383ae832b477c8129bbb4fe162a0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } } peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } }
I20240502 08:00:01.833312 2427 raft_consensus.cc:348] T de216561c2c04e3bbb8ea066078e9a40 P 2eea07a6840f4ca9997cb720a558d378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } } peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } }
I20240502 08:00:01.833320 2428 raft_consensus.cc:374] T de216561c2c04e3bbb8ea066078e9a40 P 3e4383ae832b477c8129bbb4fe162a0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:00:01.833431 2427 raft_consensus.cc:374] T de216561c2c04e3bbb8ea066078e9a40 P 2eea07a6840f4ca9997cb720a558d378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:00:01.833441 2428 raft_consensus.cc:729] T de216561c2c04e3bbb8ea066078e9a40 P 3e4383ae832b477c8129bbb4fe162a0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e4383ae832b477c8129bbb4fe162a0e, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.833485 2427 raft_consensus.cc:729] T de216561c2c04e3bbb8ea066078e9a40 P 2eea07a6840f4ca9997cb720a558d378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eea07a6840f4ca9997cb720a558d378, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.833621 2428 consensus_queue.cc:260] T de216561c2c04e3bbb8ea066078e9a40 P 3e4383ae832b477c8129bbb4fe162a0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } } peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } }
I20240502 08:00:01.833714 2427 consensus_queue.cc:260] T de216561c2c04e3bbb8ea066078e9a40 P 2eea07a6840f4ca9997cb720a558d378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } } peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } }
I20240502 08:00:01.833997 2428 ts_tablet_manager.cc:1398] T de216561c2c04e3bbb8ea066078e9a40 P 3e4383ae832b477c8129bbb4fe162a0e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:00:01.834154 2427 ts_tablet_manager.cc:1398] T de216561c2c04e3bbb8ea066078e9a40 P 2eea07a6840f4ca9997cb720a558d378: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 08:00:01.834174 2428 tablet_bootstrap.cc:492] T e2ec882e67844f4faebdedf5e64973c9 P 3e4383ae832b477c8129bbb4fe162a0e: Bootstrap starting.
I20240502 08:00:01.834323 2427 tablet_bootstrap.cc:492] T 1a2593362f42463f80ea79b73a99e592 P 2eea07a6840f4ca9997cb720a558d378: Bootstrap starting.
I20240502 08:00:01.835294 2428 tablet_bootstrap.cc:654] T e2ec882e67844f4faebdedf5e64973c9 P 3e4383ae832b477c8129bbb4fe162a0e: Neither blocks nor log segments found. Creating new log.
I20240502 08:00:01.835378 2427 tablet_bootstrap.cc:654] T 1a2593362f42463f80ea79b73a99e592 P 2eea07a6840f4ca9997cb720a558d378: Neither blocks nor log segments found. Creating new log.
I20240502 08:00:01.833323 2429 ts_tablet_manager.cc:1398] T de216561c2c04e3bbb8ea066078e9a40 P 0994655074c74efea4418291148fb7d7: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:00:01.837460 2428 tablet_bootstrap.cc:492] T e2ec882e67844f4faebdedf5e64973c9 P 3e4383ae832b477c8129bbb4fe162a0e: No bootstrap required, opened a new log
I20240502 08:00:01.837572 2428 ts_tablet_manager.cc:1367] T e2ec882e67844f4faebdedf5e64973c9 P 3e4383ae832b477c8129bbb4fe162a0e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:00:01.838104 2427 tablet_bootstrap.cc:492] T 1a2593362f42463f80ea79b73a99e592 P 2eea07a6840f4ca9997cb720a558d378: No bootstrap required, opened a new log
I20240502 08:00:01.838091 2428 raft_consensus.cc:348] T e2ec882e67844f4faebdedf5e64973c9 P 3e4383ae832b477c8129bbb4fe162a0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } } peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } }
I20240502 08:00:01.838217 2428 raft_consensus.cc:374] T e2ec882e67844f4faebdedf5e64973c9 P 3e4383ae832b477c8129bbb4fe162a0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:00:01.838243 2427 ts_tablet_manager.cc:1367] T 1a2593362f42463f80ea79b73a99e592 P 2eea07a6840f4ca9997cb720a558d378: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240502 08:00:01.838249 2428 raft_consensus.cc:729] T e2ec882e67844f4faebdedf5e64973c9 P 3e4383ae832b477c8129bbb4fe162a0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e4383ae832b477c8129bbb4fe162a0e, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.838284 2436 leader_election.cc:288] T e2ec882e67844f4faebdedf5e64973c9 P 2eea07a6840f4ca9997cb720a558d378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e4383ae832b477c8129bbb4fe162a0e (127.29.106.129:33195), 0994655074c74efea4418291148fb7d7 (127.29.106.131:33267)
I20240502 08:00:01.838894 2428 consensus_queue.cc:260] T e2ec882e67844f4faebdedf5e64973c9 P 3e4383ae832b477c8129bbb4fe162a0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } } peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } }
I20240502 08:00:01.839253 2428 ts_tablet_manager.cc:1398] T e2ec882e67844f4faebdedf5e64973c9 P 3e4383ae832b477c8129bbb4fe162a0e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:00:01.838719 2427 raft_consensus.cc:348] T 1a2593362f42463f80ea79b73a99e592 P 2eea07a6840f4ca9997cb720a558d378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } } peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } }
I20240502 08:00:01.839536 2427 raft_consensus.cc:374] T 1a2593362f42463f80ea79b73a99e592 P 2eea07a6840f4ca9997cb720a558d378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:00:01.839568 2427 raft_consensus.cc:729] T 1a2593362f42463f80ea79b73a99e592 P 2eea07a6840f4ca9997cb720a558d378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eea07a6840f4ca9997cb720a558d378, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.839676 2427 consensus_queue.cc:260] T 1a2593362f42463f80ea79b73a99e592 P 2eea07a6840f4ca9997cb720a558d378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } } peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } }
I20240502 08:00:01.839918 2427 ts_tablet_manager.cc:1398] T 1a2593362f42463f80ea79b73a99e592 P 2eea07a6840f4ca9997cb720a558d378: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:00:01.841603 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ec882e67844f4faebdedf5e64973c9" candidate_uuid: "2eea07a6840f4ca9997cb720a558d378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e4383ae832b477c8129bbb4fe162a0e" is_pre_election: true
I20240502 08:00:01.841745 2230 raft_consensus.cc:2454] T e2ec882e67844f4faebdedf5e64973c9 P 3e4383ae832b477c8129bbb4fe162a0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2eea07a6840f4ca9997cb720a558d378 in term 0.
I20240502 08:00:01.842031 2270 leader_election.cc:302] T e2ec882e67844f4faebdedf5e64973c9 P 2eea07a6840f4ca9997cb720a558d378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2eea07a6840f4ca9997cb720a558d378, 3e4383ae832b477c8129bbb4fe162a0e; no voters:
I20240502 08:00:01.842262 2436 raft_consensus.cc:2781] T e2ec882e67844f4faebdedf5e64973c9 P 2eea07a6840f4ca9997cb720a558d378 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:00:01.842419 2436 raft_consensus.cc:482] T e2ec882e67844f4faebdedf5e64973c9 P 2eea07a6840f4ca9997cb720a558d378 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:00:01.842530 2436 raft_consensus.cc:3037] T e2ec882e67844f4faebdedf5e64973c9 P 2eea07a6840f4ca9997cb720a558d378 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:00:01.843786 2436 raft_consensus.cc:504] T e2ec882e67844f4faebdedf5e64973c9 P 2eea07a6840f4ca9997cb720a558d378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } } peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } }
I20240502 08:00:01.844206 2436 leader_election.cc:288] T e2ec882e67844f4faebdedf5e64973c9 P 2eea07a6840f4ca9997cb720a558d378 [CANDIDATE]: Term 1 election: Requested vote from peers 3e4383ae832b477c8129bbb4fe162a0e (127.29.106.129:33195), 0994655074c74efea4418291148fb7d7 (127.29.106.131:33267)
I20240502 08:00:01.844439 2381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ec882e67844f4faebdedf5e64973c9" candidate_uuid: "2eea07a6840f4ca9997cb720a558d378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0994655074c74efea4418291148fb7d7" is_pre_election: true
I20240502 08:00:01.844450 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ec882e67844f4faebdedf5e64973c9" candidate_uuid: "2eea07a6840f4ca9997cb720a558d378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e4383ae832b477c8129bbb4fe162a0e"
I20240502 08:00:01.844702 2380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ec882e67844f4faebdedf5e64973c9" candidate_uuid: "2eea07a6840f4ca9997cb720a558d378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0994655074c74efea4418291148fb7d7"
I20240502 08:00:01.844724 2381 raft_consensus.cc:2454] T e2ec882e67844f4faebdedf5e64973c9 P 0994655074c74efea4418291148fb7d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2eea07a6840f4ca9997cb720a558d378 in term 0.
I20240502 08:00:01.844895 2230 raft_consensus.cc:3037] T e2ec882e67844f4faebdedf5e64973c9 P 3e4383ae832b477c8129bbb4fe162a0e [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:00:01.846150 2230 raft_consensus.cc:2454] T e2ec882e67844f4faebdedf5e64973c9 P 3e4383ae832b477c8129bbb4fe162a0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2eea07a6840f4ca9997cb720a558d378 in term 1.
I20240502 08:00:01.846469 2270 leader_election.cc:302] T e2ec882e67844f4faebdedf5e64973c9 P 2eea07a6840f4ca9997cb720a558d378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2eea07a6840f4ca9997cb720a558d378, 3e4383ae832b477c8129bbb4fe162a0e; no voters: 0994655074c74efea4418291148fb7d7
I20240502 08:00:01.846782 2436 raft_consensus.cc:2781] T e2ec882e67844f4faebdedf5e64973c9 P 2eea07a6840f4ca9997cb720a558d378 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:00:01.846900 2436 raft_consensus.cc:686] T e2ec882e67844f4faebdedf5e64973c9 P 2eea07a6840f4ca9997cb720a558d378 [term 1 LEADER]: Becoming Leader. State: Replica: 2eea07a6840f4ca9997cb720a558d378, State: Running, Role: LEADER
I20240502 08:00:01.847038 2436 consensus_queue.cc:237] T e2ec882e67844f4faebdedf5e64973c9 P 2eea07a6840f4ca9997cb720a558d378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } } peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } }
I20240502 08:00:01.848984 2126 catalog_manager.cc:5484] T e2ec882e67844f4faebdedf5e64973c9 P 2eea07a6840f4ca9997cb720a558d378 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2eea07a6840f4ca9997cb720a558d378 (127.29.106.130). New cstate: current_term: 1 leader_uuid: "2eea07a6840f4ca9997cb720a558d378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } health_report { overall_health: UNKNOWN } } }
I20240502 08:00:01.863411 2440 raft_consensus.cc:482] T 1a2593362f42463f80ea79b73a99e592 P 3e4383ae832b477c8129bbb4fe162a0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:00:01.863559 2440 raft_consensus.cc:504] T 1a2593362f42463f80ea79b73a99e592 P 3e4383ae832b477c8129bbb4fe162a0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } } peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } }
I20240502 08:00:01.864097 2440 leader_election.cc:288] T 1a2593362f42463f80ea79b73a99e592 P 3e4383ae832b477c8129bbb4fe162a0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2eea07a6840f4ca9997cb720a558d378 (127.29.106.130:45293), 0994655074c74efea4418291148fb7d7 (127.29.106.131:33267)
I20240502 08:00:01.867856 2306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a2593362f42463f80ea79b73a99e592" candidate_uuid: "3e4383ae832b477c8129bbb4fe162a0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eea07a6840f4ca9997cb720a558d378" is_pre_election: true
I20240502 08:00:01.868036 2306 raft_consensus.cc:2454] T 1a2593362f42463f80ea79b73a99e592 P 2eea07a6840f4ca9997cb720a558d378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e4383ae832b477c8129bbb4fe162a0e in term 0.
I20240502 08:00:01.868289 2381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a2593362f42463f80ea79b73a99e592" candidate_uuid: "3e4383ae832b477c8129bbb4fe162a0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0994655074c74efea4418291148fb7d7" is_pre_election: true
I20240502 08:00:01.868431 2193 leader_election.cc:302] T 1a2593362f42463f80ea79b73a99e592 P 3e4383ae832b477c8129bbb4fe162a0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2eea07a6840f4ca9997cb720a558d378, 3e4383ae832b477c8129bbb4fe162a0e; no voters:
I20240502 08:00:01.868499 2381 raft_consensus.cc:2454] T 1a2593362f42463f80ea79b73a99e592 P 0994655074c74efea4418291148fb7d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e4383ae832b477c8129bbb4fe162a0e in term 0.
I20240502 08:00:01.868688 2440 raft_consensus.cc:2781] T 1a2593362f42463f80ea79b73a99e592 P 3e4383ae832b477c8129bbb4fe162a0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:00:01.868778 2440 raft_consensus.cc:482] T 1a2593362f42463f80ea79b73a99e592 P 3e4383ae832b477c8129bbb4fe162a0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:00:01.868826 2440 raft_consensus.cc:3037] T 1a2593362f42463f80ea79b73a99e592 P 3e4383ae832b477c8129bbb4fe162a0e [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:00:01.869868 2440 raft_consensus.cc:504] T 1a2593362f42463f80ea79b73a99e592 P 3e4383ae832b477c8129bbb4fe162a0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } } peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } }
I20240502 08:00:01.870247 2440 leader_election.cc:288] T 1a2593362f42463f80ea79b73a99e592 P 3e4383ae832b477c8129bbb4fe162a0e [CANDIDATE]: Term 1 election: Requested vote from peers 2eea07a6840f4ca9997cb720a558d378 (127.29.106.130:45293), 0994655074c74efea4418291148fb7d7 (127.29.106.131:33267)
I20240502 08:00:01.870815 2381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a2593362f42463f80ea79b73a99e592" candidate_uuid: "3e4383ae832b477c8129bbb4fe162a0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0994655074c74efea4418291148fb7d7"
I20240502 08:00:01.870815 2306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a2593362f42463f80ea79b73a99e592" candidate_uuid: "3e4383ae832b477c8129bbb4fe162a0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eea07a6840f4ca9997cb720a558d378"
I20240502 08:00:01.871100 2381 raft_consensus.cc:3037] T 1a2593362f42463f80ea79b73a99e592 P 0994655074c74efea4418291148fb7d7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:00:01.871100 2306 raft_consensus.cc:3037] T 1a2593362f42463f80ea79b73a99e592 P 2eea07a6840f4ca9997cb720a558d378 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:00:01.872203 2381 raft_consensus.cc:2454] T 1a2593362f42463f80ea79b73a99e592 P 0994655074c74efea4418291148fb7d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e4383ae832b477c8129bbb4fe162a0e in term 1.
I20240502 08:00:01.872203 2306 raft_consensus.cc:2454] T 1a2593362f42463f80ea79b73a99e592 P 2eea07a6840f4ca9997cb720a558d378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e4383ae832b477c8129bbb4fe162a0e in term 1.
I20240502 08:00:01.872798 2196 leader_election.cc:302] T 1a2593362f42463f80ea79b73a99e592 P 3e4383ae832b477c8129bbb4fe162a0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0994655074c74efea4418291148fb7d7, 3e4383ae832b477c8129bbb4fe162a0e; no voters:
I20240502 08:00:01.873186 2440 raft_consensus.cc:2781] T 1a2593362f42463f80ea79b73a99e592 P 3e4383ae832b477c8129bbb4fe162a0e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:00:01.873278 2440 raft_consensus.cc:686] T 1a2593362f42463f80ea79b73a99e592 P 3e4383ae832b477c8129bbb4fe162a0e [term 1 LEADER]: Becoming Leader. State: Replica: 3e4383ae832b477c8129bbb4fe162a0e, State: Running, Role: LEADER
I20240502 08:00:01.873461 2440 consensus_queue.cc:237] T 1a2593362f42463f80ea79b73a99e592 P 3e4383ae832b477c8129bbb4fe162a0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } } peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } }
I20240502 08:00:01.875705 2127 catalog_manager.cc:5484] T 1a2593362f42463f80ea79b73a99e592 P 3e4383ae832b477c8129bbb4fe162a0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e4383ae832b477c8129bbb4fe162a0e (127.29.106.129). New cstate: current_term: 1 leader_uuid: "3e4383ae832b477c8129bbb4fe162a0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } health_report { overall_health: HEALTHY } } }
I20240502 08:00:01.898850 2432 raft_consensus.cc:482] T de216561c2c04e3bbb8ea066078e9a40 P 0994655074c74efea4418291148fb7d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:00:01.899015 2432 raft_consensus.cc:504] T de216561c2c04e3bbb8ea066078e9a40 P 0994655074c74efea4418291148fb7d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } } peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } }
I20240502 08:00:01.899533 2432 leader_election.cc:288] T de216561c2c04e3bbb8ea066078e9a40 P 0994655074c74efea4418291148fb7d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e4383ae832b477c8129bbb4fe162a0e (127.29.106.129:33195), 2eea07a6840f4ca9997cb720a558d378 (127.29.106.130:45293)
I20240502 08:00:01.903776 2306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de216561c2c04e3bbb8ea066078e9a40" candidate_uuid: "0994655074c74efea4418291148fb7d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eea07a6840f4ca9997cb720a558d378" is_pre_election: true
I20240502 08:00:01.903875 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de216561c2c04e3bbb8ea066078e9a40" candidate_uuid: "0994655074c74efea4418291148fb7d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e4383ae832b477c8129bbb4fe162a0e" is_pre_election: true
I20240502 08:00:01.904047 2230 raft_consensus.cc:2454] T de216561c2c04e3bbb8ea066078e9a40 P 3e4383ae832b477c8129bbb4fe162a0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0994655074c74efea4418291148fb7d7 in term 0.
I20240502 08:00:01.904047 2306 raft_consensus.cc:2454] T de216561c2c04e3bbb8ea066078e9a40 P 2eea07a6840f4ca9997cb720a558d378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0994655074c74efea4418291148fb7d7 in term 0.
I20240502 08:00:01.904469 2344 leader_election.cc:302] T de216561c2c04e3bbb8ea066078e9a40 P 0994655074c74efea4418291148fb7d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0994655074c74efea4418291148fb7d7, 2eea07a6840f4ca9997cb720a558d378; no voters:
I20240502 08:00:01.904685 2432 raft_consensus.cc:2781] T de216561c2c04e3bbb8ea066078e9a40 P 0994655074c74efea4418291148fb7d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:00:01.904773 2432 raft_consensus.cc:482] T de216561c2c04e3bbb8ea066078e9a40 P 0994655074c74efea4418291148fb7d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:00:01.904803 2432 raft_consensus.cc:3037] T de216561c2c04e3bbb8ea066078e9a40 P 0994655074c74efea4418291148fb7d7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:00:01.905728 2432 raft_consensus.cc:504] T de216561c2c04e3bbb8ea066078e9a40 P 0994655074c74efea4418291148fb7d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } } peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } }
I20240502 08:00:01.906111 2432 leader_election.cc:288] T de216561c2c04e3bbb8ea066078e9a40 P 0994655074c74efea4418291148fb7d7 [CANDIDATE]: Term 1 election: Requested vote from peers 3e4383ae832b477c8129bbb4fe162a0e (127.29.106.129:33195), 2eea07a6840f4ca9997cb720a558d378 (127.29.106.130:45293)
I20240502 08:00:01.906502 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de216561c2c04e3bbb8ea066078e9a40" candidate_uuid: "0994655074c74efea4418291148fb7d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e4383ae832b477c8129bbb4fe162a0e"
I20240502 08:00:01.906502 2306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de216561c2c04e3bbb8ea066078e9a40" candidate_uuid: "0994655074c74efea4418291148fb7d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eea07a6840f4ca9997cb720a558d378"
I20240502 08:00:01.906657 2230 raft_consensus.cc:3037] T de216561c2c04e3bbb8ea066078e9a40 P 3e4383ae832b477c8129bbb4fe162a0e [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:00:01.906672 2306 raft_consensus.cc:3037] T de216561c2c04e3bbb8ea066078e9a40 P 2eea07a6840f4ca9997cb720a558d378 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:00:01.907608 2306 raft_consensus.cc:2454] T de216561c2c04e3bbb8ea066078e9a40 P 2eea07a6840f4ca9997cb720a558d378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0994655074c74efea4418291148fb7d7 in term 1.
I20240502 08:00:01.907608 2230 raft_consensus.cc:2454] T de216561c2c04e3bbb8ea066078e9a40 P 3e4383ae832b477c8129bbb4fe162a0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0994655074c74efea4418291148fb7d7 in term 1.
I20240502 08:00:01.907968 2344 leader_election.cc:302] T de216561c2c04e3bbb8ea066078e9a40 P 0994655074c74efea4418291148fb7d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0994655074c74efea4418291148fb7d7, 2eea07a6840f4ca9997cb720a558d378; no voters:
I20240502 08:00:01.908136 2432 raft_consensus.cc:2781] T de216561c2c04e3bbb8ea066078e9a40 P 0994655074c74efea4418291148fb7d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:00:01.908393 2432 raft_consensus.cc:686] T de216561c2c04e3bbb8ea066078e9a40 P 0994655074c74efea4418291148fb7d7 [term 1 LEADER]: Becoming Leader. State: Replica: 0994655074c74efea4418291148fb7d7, State: Running, Role: LEADER
I20240502 08:00:01.908560 2432 consensus_queue.cc:237] T de216561c2c04e3bbb8ea066078e9a40 P 0994655074c74efea4418291148fb7d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } } peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } }
I20240502 08:00:01.910352 2127 catalog_manager.cc:5484] T de216561c2c04e3bbb8ea066078e9a40 P 0994655074c74efea4418291148fb7d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0994655074c74efea4418291148fb7d7 (127.29.106.131). New cstate: current_term: 1 leader_uuid: "0994655074c74efea4418291148fb7d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } health_report { overall_health: HEALTHY } } }
I20240502 08:00:01.928040 2436 consensus_queue.cc:1035] T e2ec882e67844f4faebdedf5e64973c9 P 2eea07a6840f4ca9997cb720a558d378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:00:01.931782 2381 raft_consensus.cc:3037] T e2ec882e67844f4faebdedf5e64973c9 P 0994655074c74efea4418291148fb7d7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:00:01.933493 2456 consensus_queue.cc:1035] T e2ec882e67844f4faebdedf5e64973c9 P 2eea07a6840f4ca9997cb720a558d378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:00:01.935737 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:00:01.938026 2257 tablet.cc:2369] T 1a2593362f42463f80ea79b73a99e592 P 3e4383ae832b477c8129bbb4fe162a0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 08:00:01.939122 2461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:00:01.939976 2460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:00:01.940691 2463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:00:01.941498 30122 server_base.cc:1034] running on GCE node
I20240502 08:00:01.941726 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:00:01.941774 30122 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:00:01.941800 30122 hybrid_clock.cc:648] HybridClock initialized: now 1714636801941799 us; error 0 us; skew 500 ppm
I20240502 08:00:01.942978 30122 webserver.cc:460] Webserver started at http://127.29.106.132:42377/ using document root <none> and password file <none>
I20240502 08:00:01.943161 30122 fs_manager.cc:359] Metadata directory not provided
I20240502 08:00:01.943217 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:00:01.943297 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:00:01.943730 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636688849691-30122-0/minicluster-data/ts-3-root/instance:
uuid: "acd6bc291c944854a05d28c67f0d11b8"
format_stamp: "Formatted at 2024-05-02 08:00:01 on dist-test-slave-m6l2"
I20240502 08:00:01.945627 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 08:00:01.946861 2468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:00:01.947274 30122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 08:00:01.947470 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636688849691-30122-0/minicluster-data/ts-3-root
uuid: "acd6bc291c944854a05d28c67f0d11b8"
format_stamp: "Formatted at 2024-05-02 08:00:01 on dist-test-slave-m6l2"
I20240502 08:00:01.947571 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636688849691-30122-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636688849691-30122-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskvYI9Bw/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636688849691-30122-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:00:01.968353 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:00:01.968796 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:00:01.969115 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:00:01.969584 30122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:00:01.969627 30122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:00:01.969686 30122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:00:01.969720 30122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:00:01.973076 2440 consensus_queue.cc:1035] T 1a2593362f42463f80ea79b73a99e592 P 3e4383ae832b477c8129bbb4fe162a0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:00:01.976151 30122 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.132:42717
W20240502 08:00:01.976451 2437 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20240502 08:00:01.976523 2437 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20240502 08:00:01.976958 2532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.132:42717 every 8 connection(s)
I20240502 08:00:01.980973 2440 consensus_queue.cc:1035] T 1a2593362f42463f80ea79b73a99e592 P 3e4383ae832b477c8129bbb4fe162a0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:00:01.982729 2533 heartbeater.cc:347] Connected to a master server at 127.29.106.190:45161
I20240502 08:00:01.982825 2533 heartbeater.cc:464] Registering TS with master...
I20240502 08:00:01.983093 2533 heartbeater.cc:511] Master 127.29.106.190:45161 requested a full tablet report, sending...
I20240502 08:00:01.983712 2126 ts_manager.cc:194] Registered new tserver with Master: acd6bc291c944854a05d28c67f0d11b8 (127.29.106.132:42717)
I20240502 08:00:01.984381 2126 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42924
I20240502 08:00:01.986172 2432 consensus_queue.cc:1035] T de216561c2c04e3bbb8ea066078e9a40 P 0994655074c74efea4418291148fb7d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:00:02.014413 2432 consensus_queue.cc:1035] T de216561c2c04e3bbb8ea066078e9a40 P 0994655074c74efea4418291148fb7d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:00:02.621510 2181 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:00:02.622095 2181 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:00:02.622184 2181 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:00:02.627671 2381 consensus_queue.cc:237] T de216561c2c04e3bbb8ea066078e9a40 P 0994655074c74efea4418291148fb7d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } } peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } attrs { replace: true } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } } peers { permanent_uuid: "acd6bc291c944854a05d28c67f0d11b8" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 42717 } attrs { promote: true } }
I20240502 08:00:02.629298 2306 raft_consensus.cc:1261] T de216561c2c04e3bbb8ea066078e9a40 P 2eea07a6840f4ca9997cb720a558d378 [term 1 FOLLOWER]: Refusing update from remote peer 0994655074c74efea4418291148fb7d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:00:02.629343 2230 raft_consensus.cc:1261] T de216561c2c04e3bbb8ea066078e9a40 P 3e4383ae832b477c8129bbb4fe162a0e [term 1 FOLLOWER]: Refusing update from remote peer 0994655074c74efea4418291148fb7d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:00:02.629745 2543 consensus_queue.cc:1035] T de216561c2c04e3bbb8ea066078e9a40 P 0994655074c74efea4418291148fb7d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:00:02.629976 2452 consensus_queue.cc:1035] T de216561c2c04e3bbb8ea066078e9a40 P 0994655074c74efea4418291148fb7d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:00:02.631922 2543 raft_consensus.cc:2932] T de216561c2c04e3bbb8ea066078e9a40 P 0994655074c74efea4418291148fb7d7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER acd6bc291c944854a05d28c67f0d11b8 (127.29.106.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } } peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } attrs { replace: true } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } } peers { permanent_uuid: "acd6bc291c944854a05d28c67f0d11b8" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 42717 } attrs { promote: true } } }
I20240502 08:00:02.632442 2306 raft_consensus.cc:2932] T de216561c2c04e3bbb8ea066078e9a40 P 2eea07a6840f4ca9997cb720a558d378 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER acd6bc291c944854a05d28c67f0d11b8 (127.29.106.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } } peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } attrs { replace: true } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } } peers { permanent_uuid: "acd6bc291c944854a05d28c67f0d11b8" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 42717 } attrs { promote: true } } }
I20240502 08:00:02.632462 2230 raft_consensus.cc:2932] T de216561c2c04e3bbb8ea066078e9a40 P 3e4383ae832b477c8129bbb4fe162a0e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER acd6bc291c944854a05d28c67f0d11b8 (127.29.106.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } } peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } attrs { replace: true } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } } peers { permanent_uuid: "acd6bc291c944854a05d28c67f0d11b8" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 42717 } attrs { promote: true } } }
W20240502 08:00:02.633754 2346 consensus_peers.cc:485] T de216561c2c04e3bbb8ea066078e9a40 P 0994655074c74efea4418291148fb7d7 -> Peer acd6bc291c944854a05d28c67f0d11b8 (127.29.106.132:42717): Couldn't send request to peer acd6bc291c944854a05d28c67f0d11b8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de216561c2c04e3bbb8ea066078e9a40. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:02.635298 2126 catalog_manager.cc:5484] T de216561c2c04e3bbb8ea066078e9a40 P 0994655074c74efea4418291148fb7d7 reported cstate change: config changed from index -1 to 2, NON_VOTER acd6bc291c944854a05d28c67f0d11b8 (127.29.106.132) added. New cstate: current_term: 1 leader_uuid: "0994655074c74efea4418291148fb7d7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acd6bc291c944854a05d28c67f0d11b8" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 42717 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:00:02.639583 2230 consensus_queue.cc:237] T 1a2593362f42463f80ea79b73a99e592 P 3e4383ae832b477c8129bbb4fe162a0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } attrs { replace: true } } peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } } peers { permanent_uuid: "acd6bc291c944854a05d28c67f0d11b8" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 42717 } attrs { promote: true } }
I20240502 08:00:02.641184 2306 raft_consensus.cc:1261] T 1a2593362f42463f80ea79b73a99e592 P 2eea07a6840f4ca9997cb720a558d378 [term 1 FOLLOWER]: Refusing update from remote peer 3e4383ae832b477c8129bbb4fe162a0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:00:02.641387 2381 raft_consensus.cc:1261] T 1a2593362f42463f80ea79b73a99e592 P 0994655074c74efea4418291148fb7d7 [term 1 FOLLOWER]: Refusing update from remote peer 3e4383ae832b477c8129bbb4fe162a0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:00:02.641630 2549 consensus_queue.cc:1035] T 1a2593362f42463f80ea79b73a99e592 P 3e4383ae832b477c8129bbb4fe162a0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:00:02.641819 2539 consensus_queue.cc:1035] T 1a2593362f42463f80ea79b73a99e592 P 3e4383ae832b477c8129bbb4fe162a0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:00:02.644096 2381 raft_consensus.cc:2932] T 1a2593362f42463f80ea79b73a99e592 P 0994655074c74efea4418291148fb7d7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER acd6bc291c944854a05d28c67f0d11b8 (127.29.106.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } attrs { replace: true } } peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } } peers { permanent_uuid: "acd6bc291c944854a05d28c67f0d11b8" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 42717 } attrs { promote: true } } }
I20240502 08:00:02.644475 2306 raft_consensus.cc:2932] T 1a2593362f42463f80ea79b73a99e592 P 2eea07a6840f4ca9997cb720a558d378 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER acd6bc291c944854a05d28c67f0d11b8 (127.29.106.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } attrs { replace: true } } peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } } peers { permanent_uuid: "acd6bc291c944854a05d28c67f0d11b8" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 42717 } attrs { promote: true } } }
W20240502 08:00:02.645143 2195 consensus_peers.cc:485] T 1a2593362f42463f80ea79b73a99e592 P 3e4383ae832b477c8129bbb4fe162a0e -> Peer acd6bc291c944854a05d28c67f0d11b8 (127.29.106.132:42717): Couldn't send request to peer acd6bc291c944854a05d28c67f0d11b8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1a2593362f42463f80ea79b73a99e592. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:02.686879 2125 catalog_manager.cc:5484] T 1a2593362f42463f80ea79b73a99e592 P 0994655074c74efea4418291148fb7d7 reported cstate change: config changed from index -1 to 2, NON_VOTER acd6bc291c944854a05d28c67f0d11b8 (127.29.106.132) added. New cstate: current_term: 1 leader_uuid: "3e4383ae832b477c8129bbb4fe162a0e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } attrs { replace: true } } peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } } peers { permanent_uuid: "acd6bc291c944854a05d28c67f0d11b8" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 42717 } attrs { promote: true } } }
I20240502 08:00:02.730031 2549 raft_consensus.cc:2932] T 1a2593362f42463f80ea79b73a99e592 P 3e4383ae832b477c8129bbb4fe162a0e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER acd6bc291c944854a05d28c67f0d11b8 (127.29.106.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2eea07a6840f4ca9997cb720a558d378" member_type: VOTER last_known_addr { host: "127.29.106.130" port: 45293 } } peers { permanent_uuid: "0994655074c74efea4418291148fb7d7" member_type: VOTER last_known_addr { host: "127.29.106.131" port: 33267 } attrs { replace: true } } peers { permanent_uuid: "3e4383ae832b477c8129bbb4fe162a0e" member_type: VOTER last_known_addr { host: "127.29.106.129" port: 33195 } } peers { permanent_uuid: "acd6bc291c944854a05d28c67f0d11b8" member_type: NON_VOTER last_known_addr { host: "127.29.106.132" port: 42717 } attrs { promote: true } } }
I20240502 08:00:02.739981 2554 ts_tablet_manager.cc:902] T 1a2593362f42463f80ea79b73a99e592 P acd6bc291c944854a05d28c67f0d11b8: Initiating tablet copy from peer 3e4383ae832b477c8129bbb4fe162a0e (127.29.106.129:33195)
F20240502 08:00:02.740288 2554 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-1a2593362f42463f80ea79b73a99e592.queue_time_us, tablet-download-pool-1a2593362f42463f80ea79b73a99e592.queue_time_us) (tablet-download-pool-2c1b6523d6724a02acc9ae19f3f0255e.queue_time_us, tablet-download-pool-2c1b6523d6724a02acc9ae19f3f0255e.queue_time_us) (tablet-download-pool-2c1b6523d6724a02acc9ae19f3f0255e.run_cpu_time_us, tablet-download-pool-2c1b6523d6724a02acc9ae19f3f0255e.run_cpu_time_us) (tablet-download-pool-2c1b6523d6724a02acc9ae19f3f0255e.run_wall_time_us, tablet-download-pool-2c1b6523d6724a02acc9ae19f3f0255e.run_wall_time_us) (tablet-download-pool-2ecebe8dad3946bd956afacf695fabaa.queue_time_us, tablet-download-pool-2ecebe8dad3946bd956afacf695fabaa.queue_time_us) (tablet-download-pool-2ecebe8dad3946bd956afacf695fabaa.run_cpu_time_us, tablet-download-pool-2ecebe8dad3946bd956afacf695fabaa.run_cpu_time_us) (tablet-download-pool-2ecebe8dad3946bd956afacf695fabaa.run_wall_time_us, tablet-download-pool-2ecebe8dad3946bd956afacf695fabaa.run_wall_time_us) (tablet-download-pool-5437f94679a841e3980053e69d624d76.queue_time_us, tablet-download-pool-5437f94679a841e3980053e69d624d76.queue_time_us) (tablet-download-pool-5437f94679a841e3980053e69d624d76.run_cpu_time_us, tablet-download-pool-5437f94679a841e3980053e69d624d76.run_cpu_time_us) (tablet-download-pool-5437f94679a841e3980053e69d624d76.run_wall_time_us, tablet-download-pool-5437f94679a841e3980053e69d624d76.run_wall_time_us) (tablet-download-pool-58c5f52a420c467684b2cb50972e7c99.queue_time_us, tablet-download-pool-58c5f52a420c467684b2cb50972e7c99.queue_time_us) (tablet-download-pool-58c5f52a420c467684b2cb50972e7c99.run_cpu_time_us, tablet-download-pool-58c5f52a420c467684b2cb50972e7c99.run_cpu_time_us) (tablet-download-pool-58c5f52a420c467684b2cb50972e7c99.run_wall_time_us, tablet-download-pool-58c5f52a420c467684b2cb50972e7c99.run_wall_time_us) (tablet-download-pool-682e26b333104f82863b92c545cfdafe.queue_time_us, tablet-download-pool-682e26b333104f82863b92c545cfdafe.queue_time_us) (tablet-download-pool-682e26b333104f82863b92c545cfdafe.run_cpu_time_us, tablet-download-pool-682e26b333104f82863b92c545cfdafe.run_cpu_time_us) (tablet-download-pool-682e26b333104f82863b92c545cfdafe.run_wall_time_us, tablet-download-pool-682e26b333104f82863b92c545cfdafe.run_wall_time_us) (tablet-download-pool-69a9ebd0c390436b8ace08089f719358.queue_time_us, tablet-download-pool-69a9ebd0c390436b8ace08089f719358.queue_time_us) (tablet-download-pool-69a9ebd0c390436b8ace08089f719358.run_cpu_time_us, tablet-download-pool-69a9ebd0c390436b8ace08089f719358.run_cpu_time_us) (tablet-download-pool-69a9ebd0c390436b8ace08089f719358.run_wall_time_us, tablet-download-pool-69a9ebd0c390436b8ace08089f719358.run_wall_time_us) (tablet-download-pool-6edf17550e3f4a0b945477edc6de6af1.queue_time_us, tablet-download-pool-6edf17550e3f4a0b945477edc6de6af1.queue_time_us) (tablet-download-pool-6edf17550e3f4a0b945477edc6de6af1.run_cpu_time_us, tablet-download-pool-6edf17550e3f4a0b945477edc6de6af1.run_cpu_time_us) (tablet-download-pool-6edf17550e3f4a0b945477edc6de6af1.run_wall_time_us, tablet-download-pool-6edf17550e3f4a0b945477edc6de6af1.run_wall_time_us) (tablet-download-pool-862e4df4c8004aa0973a06875b587e5f.queue_time_us, tablet-download-pool-862e4df4c8004aa0973a06875b587e5f.queue_time_us) (tablet-download-pool-862e4df4c8004aa0973a06875b587e5f.run_cpu_time_us, tablet-download-pool-862e4df4c8004aa0973a06875b587e5f.run_cpu_time_us) (tablet-download-pool-862e4df4c8004aa0973a06875b587e5f.run_wall_time_us, tablet-download-pool-862e4df4c8004aa0973a06875b587e5f.run_wall_time_us) (tablet-download-pool-c7c1536618ae4e74aa79349d04201fb9.queue_time_us, tablet-download-pool-c7c1536618ae4e74aa79349d04201fb9.queue_time_us) (tablet-download-pool-c7c1536618ae4e74aa79349d04201fb9.run_cpu_time_us, tablet-download-pool-c7c1536618ae4e74aa79349d04201fb9.run_cpu_time_us) (tablet-download-pool-c7c1536618ae4e74aa79349d04201fb9.run_wall_time_us, tablet-download-pool-c7c1536618ae4e74aa79349d04201fb9.run_wall_time_us) (tablet-download-pool-cdf0ff2967c040a589f915172d3f446b.queue_time_us, tablet-download-pool-cdf0ff2967c040a589f915172d3f446b.queue_time_us) (tablet-download-pool-cdf0ff2967c040a589f915172d3f446b.run_cpu_time_us, tablet-download-pool-cdf0ff2967c040a589f915172d3f446b.run_cpu_time_us) (tablet-download-pool-cdf0ff2967c040a589f915172d3f446b.run_wall_time_us, tablet-download-pool-cdf0ff2967c040a589f915172d3f446b.run_wall_time_us) (tablet-download-pool-df4233b549b34b01b6f9662c7e7b3515.queue_time_us, tablet-download-pool-df4233b549b34b01b6f9662c7e7b3515.queue_time_us) (tablet-download-pool-df4233b549b34b01b6f9662c7e7b3515.run_cpu_time_us, tablet-download-pool-df4233b549b34b01b6f9662c7e7b3515.run_cpu_time_us) (tablet-download-pool-df4233b549b34b01b6f9662c7e7b3515.run_wall_time_us, tablet-download-pool-df4233b549b34b01b6f9662c7e7b3515.run_wall_time_us) (tablet-download-pool-fda98cfe14f947969fb41e26c75bc384.queue_time_us, tablet-download-pool-fda98cfe14f947969fb41e26c75bc384.queue_time_us) (tablet-download-pool-fda98cfe14f947969fb41e26c75bc384.run_cpu_time_us, tablet-download-pool-fda98cfe14f947969fb41e26c75bc384.run_cpu_time_us) (tablet-download-pool-fda98cfe14f947969fb41e26c75bc384.run_wall_time_us, tablet-download-pool-fda98cfe14f947969fb41e26c75bc384.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
*** Aborted at 1714636802 (unix time) try "date -d @1714636802" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000075aa) received by PID 30122 (TID 0x7fe9d9e73700) from PID 30122; stack trace: ***
@ 0x7fea30065040 (unknown) at ??:0
I20240502 08:00:02.751333 2556 ts_tablet_manager.cc:902] T de216561c2c04e3bbb8ea066078e9a40 P acd6bc291c944854a05d28c67f0d11b8: Initiating tablet copy from peer 0994655074c74efea4418291148fb7d7 (127.29.106.131:33267)
@ 0x7fea30064fb7 gsignal at ??:0
@ 0x7fea30066921 abort at ??:0
@ 0x7fea30d69dcd google::LogMessage::Fail() at ??:0
@ 0x7fea30d6db93 google::LogMessage::SendToLog() at ??:0
@ 0x7fea30d697cc google::LogMessage::Flush() at ??:0
@ 0x7fea30d6af59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fea318c1044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fea318a5f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fea318a3efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fea3282077a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fea32829a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fea328aa76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fea328be4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fea328a9226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7fea328b8c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fea32e133cc std::function<>::operator()() at ??:0
@ 0x7fea318a8e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fea318a96ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fea318ab03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fea32e133cc std::function<>::operator()() at ??:0
@ 0x7fea31898807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fea2e09a6db start_thread at ??:0
@ 0x7fea3014771f clone at ??:0