[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 13:59:36.540853 581 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.145.126:39243,127.0.145.125:46463,127.0.145.124:41133
I20240502 13:59:36.541738 581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:36.542213 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:36.549124 586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:36.549531 589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:36.549868 587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:36.550230 581 server_base.cc:1034] running on GCE node
I20240502 13:59:36.550647 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 13:59:36.550748 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 13:59:36.550784 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658376550782 us; error 0 us; skew 500 ppm
I20240502 13:59:36.552446 581 webserver.cc:460] Webserver started at http://127.0.145.126:34109/ using document root <none> and password file <none>
I20240502 13:59:36.552942 581 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:36.553000 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:36.553189 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:36.554723 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658376531600-581-0/minicluster-data/master-0-root/instance:
uuid: "793b98f34d2e4038917c8b6d1e5e4a8b"
format_stamp: "Formatted at 2024-05-02 13:59:36 on dist-test-slave-1z5j"
I20240502 13:59:36.557790 581 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 13:59:36.559641 594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:36.560565 581 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:36.560664 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658376531600-581-0/minicluster-data/master-0-root
uuid: "793b98f34d2e4038917c8b6d1e5e4a8b"
format_stamp: "Formatted at 2024-05-02 13:59:36 on dist-test-slave-1z5j"
I20240502 13:59:36.560736 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658376531600-581-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658376531600-581-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658376531600-581-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:36.610045 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:36.610639 581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:36.610785 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:36.616925 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.126:39243
I20240502 13:59:36.616950 646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.126:39243 every 8 connection(s)
I20240502 13:59:36.617945 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:36.619536 649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:36.619920 652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:36.620069 647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 13:59:36.620229 650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:36.620610 581 server_base.cc:1034] running on GCE node
I20240502 13:59:36.620792 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 13:59:36.620838 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 13:59:36.620863 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658376620863 us; error 0 us; skew 500 ppm
I20240502 13:59:36.621698 581 webserver.cc:460] Webserver started at http://127.0.145.125:34195/ using document root <none> and password file <none>
I20240502 13:59:36.621831 581 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:36.621879 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:36.621945 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:36.622313 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658376531600-581-0/minicluster-data/master-1-root/instance:
uuid: "2d0dd09a40324e1c8cb7dc95b25f6375"
format_stamp: "Formatted at 2024-05-02 13:59:36 on dist-test-slave-1z5j"
I20240502 13:59:36.623912 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240502 13:59:36.624871 657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:36.625118 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:36.625175 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658376531600-581-0/minicluster-data/master-1-root
uuid: "2d0dd09a40324e1c8cb7dc95b25f6375"
format_stamp: "Formatted at 2024-05-02 13:59:36 on dist-test-slave-1z5j"
I20240502 13:59:36.625231 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658376531600-581-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658376531600-581-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658376531600-581-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:36.625428 647 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.145.126" port: 39243 } has no permanent_uuid. Determining permanent_uuid...
I20240502 13:59:36.634279 647 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.145.125" port: 46463 } has no permanent_uuid. Determining permanent_uuid...
W20240502 13:59:36.634850 597 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.125:46463: connect: Connection refused (error 111)
I20240502 13:59:36.636127 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
W20240502 13:59:36.636278 647 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.145.125:46463: Network error: Client connection negotiation failed: client connection to 127.0.145.125:46463: connect: Connection refused (error 111)
I20240502 13:59:36.636436 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:36.640674 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.125:46463
I20240502 13:59:36.640725 713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.125:46463 every 8 connection(s)
I20240502 13:59:36.641274 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 13:59:36.641427 714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:36.642457 714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.145.126" port: 39243 } has no permanent_uuid. Determining permanent_uuid...
W20240502 13:59:36.642648 716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:36.643741 717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:36.644030 719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:36.644410 581 server_base.cc:1034] running on GCE node
I20240502 13:59:36.644560 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 13:59:36.644595 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 13:59:36.644614 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658376644613 us; error 0 us; skew 500 ppm
I20240502 13:59:36.645447 581 webserver.cc:460] Webserver started at http://127.0.145.124:34343/ using document root <none> and password file <none>
I20240502 13:59:36.645583 581 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:36.645627 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:36.645675 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:36.646029 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658376531600-581-0/minicluster-data/master-2-root/instance:
uuid: "4008e3433a3941df8f20e1059c4ed97d"
format_stamp: "Formatted at 2024-05-02 13:59:36 on dist-test-slave-1z5j"
I20240502 13:59:36.646477 714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.145.125" port: 46463 } has no permanent_uuid. Determining permanent_uuid...
I20240502 13:59:36.647464 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 13:59:36.648272 726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:36.648510 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:36.648567 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658376531600-581-0/minicluster-data/master-2-root
uuid: "4008e3433a3941df8f20e1059c4ed97d"
format_stamp: "Formatted at 2024-05-02 13:59:36 on dist-test-slave-1z5j"
I20240502 13:59:36.648619 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658376531600-581-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658376531600-581-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658376531600-581-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:36.650008 714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.145.124" port: 41133 } has no permanent_uuid. Determining permanent_uuid...
W20240502 13:59:36.650983 714 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.145.124:41133: Network error: Client connection negotiation failed: client connection to 127.0.145.124:41133: connect: Connection refused (error 111)
I20240502 13:59:36.665438 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:36.665709 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:36.669795 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.124:41133
I20240502 13:59:36.669826 779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.124:41133 every 8 connection(s)
I20240502 13:59:36.670146 581 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 13:59:36.670481 780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:36.671329 780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.145.126" port: 39243 } has no permanent_uuid. Determining permanent_uuid...
I20240502 13:59:36.674503 780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.145.125" port: 46463 } has no permanent_uuid. Determining permanent_uuid...
I20240502 13:59:36.677353 780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.145.124" port: 41133 } has no permanent_uuid. Determining permanent_uuid...
I20240502 13:59:36.682260 780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4008e3433a3941df8f20e1059c4ed97d: Bootstrap starting.
I20240502 13:59:36.684357 780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4008e3433a3941df8f20e1059c4ed97d: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:36.685137 780 log.cc:824] T 00000000000000000000000000000000 P 4008e3433a3941df8f20e1059c4ed97d: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:36.685428 647 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.145.125" port: 46463 } attempt: 1
I20240502 13:59:36.686638 780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4008e3433a3941df8f20e1059c4ed97d: No bootstrap required, opened a new log
I20240502 13:59:36.688727 647 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.145.124" port: 41133 } has no permanent_uuid. Determining permanent_uuid...
I20240502 13:59:36.689675 780 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4008e3433a3941df8f20e1059c4ed97d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793b98f34d2e4038917c8b6d1e5e4a8b" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 39243 } } peers { permanent_uuid: "2d0dd09a40324e1c8cb7dc95b25f6375" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 46463 } } peers { permanent_uuid: "4008e3433a3941df8f20e1059c4ed97d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 41133 } }
I20240502 13:59:36.689847 780 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4008e3433a3941df8f20e1059c4ed97d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:36.689916 780 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4008e3433a3941df8f20e1059c4ed97d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4008e3433a3941df8f20e1059c4ed97d, State: Initialized, Role: FOLLOWER
I20240502 13:59:36.690265 780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4008e3433a3941df8f20e1059c4ed97d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793b98f34d2e4038917c8b6d1e5e4a8b" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 39243 } } peers { permanent_uuid: "2d0dd09a40324e1c8cb7dc95b25f6375" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 46463 } } peers { permanent_uuid: "4008e3433a3941df8f20e1059c4ed97d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 41133 } }
I20240502 13:59:36.690675 785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4008e3433a3941df8f20e1059c4ed97d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793b98f34d2e4038917c8b6d1e5e4a8b" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 39243 } } peers { permanent_uuid: "2d0dd09a40324e1c8cb7dc95b25f6375" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 46463 } } peers { permanent_uuid: "4008e3433a3941df8f20e1059c4ed97d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 41133 } } }
I20240502 13:59:36.690783 785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4008e3433a3941df8f20e1059c4ed97d [sys.catalog]: This master's current role is: FOLLOWER
I20240502 13:59:36.691346 780 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4008e3433a3941df8f20e1059c4ed97d [sys.catalog]: configured and running, proceeding with master startup.
I20240502 13:59:36.693192 647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 793b98f34d2e4038917c8b6d1e5e4a8b: Bootstrap starting.
I20240502 13:59:36.694128 647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 793b98f34d2e4038917c8b6d1e5e4a8b: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:36.694989 647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 793b98f34d2e4038917c8b6d1e5e4a8b: No bootstrap required, opened a new log
I20240502 13:59:36.695327 647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 793b98f34d2e4038917c8b6d1e5e4a8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793b98f34d2e4038917c8b6d1e5e4a8b" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 39243 } } peers { permanent_uuid: "2d0dd09a40324e1c8cb7dc95b25f6375" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 46463 } } peers { permanent_uuid: "4008e3433a3941df8f20e1059c4ed97d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 41133 } }
I20240502 13:59:36.695406 647 raft_consensus.cc:374] T 00000000000000000000000000000000 P 793b98f34d2e4038917c8b6d1e5e4a8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:36.695434 647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 793b98f34d2e4038917c8b6d1e5e4a8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793b98f34d2e4038917c8b6d1e5e4a8b, State: Initialized, Role: FOLLOWER
I20240502 13:59:36.695588 647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 793b98f34d2e4038917c8b6d1e5e4a8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793b98f34d2e4038917c8b6d1e5e4a8b" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 39243 } } peers { permanent_uuid: "2d0dd09a40324e1c8cb7dc95b25f6375" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 46463 } } peers { permanent_uuid: "4008e3433a3941df8f20e1059c4ed97d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 41133 } }
W20240502 13:59:36.695729 796 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4008e3433a3941df8f20e1059c4ed97d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 13:59:36.695801 796 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 13:59:36.695922 799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 793b98f34d2e4038917c8b6d1e5e4a8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793b98f34d2e4038917c8b6d1e5e4a8b" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 39243 } } peers { permanent_uuid: "2d0dd09a40324e1c8cb7dc95b25f6375" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 46463 } } peers { permanent_uuid: "4008e3433a3941df8f20e1059c4ed97d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 41133 } } }
I20240502 13:59:36.696024 647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 793b98f34d2e4038917c8b6d1e5e4a8b [sys.catalog]: configured and running, proceeding with master startup.
I20240502 13:59:36.696075 799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 793b98f34d2e4038917c8b6d1e5e4a8b [sys.catalog]: This master's current role is: FOLLOWER
I20240502 13:59:36.697542 581 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20240502 13:59:36.697978 810 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 793b98f34d2e4038917c8b6d1e5e4a8b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 13:59:36.698029 810 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 13:59:36.703106 714 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.145.124" port: 41133 } attempt: 1
I20240502 13:59:36.707320 714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375: Bootstrap starting.
I20240502 13:59:36.708251 714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:36.709332 714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375: No bootstrap required, opened a new log
I20240502 13:59:36.709751 714 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793b98f34d2e4038917c8b6d1e5e4a8b" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 39243 } } peers { permanent_uuid: "2d0dd09a40324e1c8cb7dc95b25f6375" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 46463 } } peers { permanent_uuid: "4008e3433a3941df8f20e1059c4ed97d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 41133 } }
I20240502 13:59:36.709851 714 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:36.709889 714 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d0dd09a40324e1c8cb7dc95b25f6375, State: Initialized, Role: FOLLOWER
I20240502 13:59:36.710005 714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793b98f34d2e4038917c8b6d1e5e4a8b" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 39243 } } peers { permanent_uuid: "2d0dd09a40324e1c8cb7dc95b25f6375" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 46463 } } peers { permanent_uuid: "4008e3433a3941df8f20e1059c4ed97d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 41133 } }
I20240502 13:59:36.710450 714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 13:59:36.710434 812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793b98f34d2e4038917c8b6d1e5e4a8b" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 39243 } } peers { permanent_uuid: "2d0dd09a40324e1c8cb7dc95b25f6375" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 46463 } } peers { permanent_uuid: "4008e3433a3941df8f20e1059c4ed97d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 41133 } } }
I20240502 13:59:36.710520 812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 13:59:36.712239 581 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20240502 13:59:36.712756 824 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20240502 13:59:36.712802 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:36.712874 824 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 13:59:36.714911 826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:36.715031 825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:36.715371 828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:36.715761 581 server_base.cc:1034] running on GCE node
I20240502 13:59:36.715914 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 13:59:36.715950 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 13:59:36.715970 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658376715968 us; error 0 us; skew 500 ppm
I20240502 13:59:36.716837 581 webserver.cc:460] Webserver started at http://127.0.145.65:40957/ using document root <none> and password file <none>
I20240502 13:59:36.716960 581 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:36.717002 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:36.717068 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:36.717371 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658376531600-581-0/minicluster-data/ts-0-root/instance:
uuid: "46b2b951921d44d1a8323c7a250e6c7d"
format_stamp: "Formatted at 2024-05-02 13:59:36 on dist-test-slave-1z5j"
I20240502 13:59:36.718560 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 13:59:36.719363 833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:36.719610 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:36.719672 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658376531600-581-0/minicluster-data/ts-0-root
uuid: "46b2b951921d44d1a8323c7a250e6c7d"
format_stamp: "Formatted at 2024-05-02 13:59:36 on dist-test-slave-1z5j"
I20240502 13:59:36.719728 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658376531600-581-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658376531600-581-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658376531600-581-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:36.739677 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:36.739993 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:36.740342 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:36.741055 581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:36.741104 581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:36.741148 581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:36.741174 581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:36.746531 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.65:39525
I20240502 13:59:36.746577 899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.65:39525 every 8 connection(s)
I20240502 13:59:36.747510 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:36.750306 906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:36.750520 907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:36.750964 909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:36.751219 581 server_base.cc:1034] running on GCE node
I20240502 13:59:36.751331 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 13:59:36.751364 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 13:59:36.751384 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658376751383 us; error 0 us; skew 500 ppm
I20240502 13:59:36.752220 900 heartbeater.cc:347] Connected to a master server at 127.0.145.124:41133
I20240502 13:59:36.752336 900 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:36.752893 900 heartbeater.cc:511] Master 127.0.145.124:41133 requested a full tablet report, sending...
I20240502 13:59:36.752877 581 webserver.cc:460] Webserver started at http://127.0.145.66:34073/ using document root <none> and password file <none>
I20240502 13:59:36.753681 581 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:36.753732 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:36.753804 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:36.754204 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658376531600-581-0/minicluster-data/ts-1-root/instance:
uuid: "0a4408b75e284c6580624cd2a730d82f"
format_stamp: "Formatted at 2024-05-02 13:59:36 on dist-test-slave-1z5j"
I20240502 13:59:36.754547 744 ts_manager.cc:194] Registered new tserver with Master: 46b2b951921d44d1a8323c7a250e6c7d (127.0.145.65:39525)
I20240502 13:59:36.754720 902 heartbeater.cc:347] Connected to a master server at 127.0.145.125:46463
I20240502 13:59:36.754786 902 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:36.754949 902 heartbeater.cc:511] Master 127.0.145.125:46463 requested a full tablet report, sending...
I20240502 13:59:36.755515 678 ts_manager.cc:194] Registered new tserver with Master: 46b2b951921d44d1a8323c7a250e6c7d (127.0.145.65:39525)
I20240502 13:59:36.755795 901 heartbeater.cc:347] Connected to a master server at 127.0.145.126:39243
I20240502 13:59:36.755865 901 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:36.755967 581 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240502 13:59:36.756047 901 heartbeater.cc:511] Master 127.0.145.126:39243 requested a full tablet report, sending...
I20240502 13:59:36.756637 611 ts_manager.cc:194] Registered new tserver with Master: 46b2b951921d44d1a8323c7a250e6c7d (127.0.145.65:39525)
I20240502 13:59:36.756906 915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:36.757145 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:36.757216 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658376531600-581-0/minicluster-data/ts-1-root
uuid: "0a4408b75e284c6580624cd2a730d82f"
format_stamp: "Formatted at 2024-05-02 13:59:36 on dist-test-slave-1z5j"
I20240502 13:59:36.757274 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658376531600-581-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658376531600-581-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658376531600-581-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:36.767683 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:36.768035 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:36.768303 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:36.768812 581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:36.768860 581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:36.768899 581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:36.768929 581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:36.774533 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.66:34109
I20240502 13:59:36.774583 979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.66:34109 every 8 connection(s)
I20240502 13:59:36.775455 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:36.777698 986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:36.780247 987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:36.780392 981 heartbeater.cc:347] Connected to a master server at 127.0.145.126:39243
I20240502 13:59:36.780472 981 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:36.780663 981 heartbeater.cc:511] Master 127.0.145.126:39243 requested a full tablet report, sending...
I20240502 13:59:36.780894 980 heartbeater.cc:347] Connected to a master server at 127.0.145.124:41133
I20240502 13:59:36.780966 980 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:36.781152 980 heartbeater.cc:511] Master 127.0.145.124:41133 requested a full tablet report, sending...
I20240502 13:59:36.781312 982 heartbeater.cc:347] Connected to a master server at 127.0.145.125:46463
I20240502 13:59:36.781395 982 heartbeater.cc:464] Registering TS with master...
W20240502 13:59:36.781575 989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:36.781617 982 heartbeater.cc:511] Master 127.0.145.125:46463 requested a full tablet report, sending...
I20240502 13:59:36.781832 611 ts_manager.cc:194] Registered new tserver with Master: 0a4408b75e284c6580624cd2a730d82f (127.0.145.66:34109)
I20240502 13:59:36.782166 678 ts_manager.cc:194] Registered new tserver with Master: 0a4408b75e284c6580624cd2a730d82f (127.0.145.66:34109)
I20240502 13:59:36.782310 744 ts_manager.cc:194] Registered new tserver with Master: 0a4408b75e284c6580624cd2a730d82f (127.0.145.66:34109)
I20240502 13:59:36.782727 581 server_base.cc:1034] running on GCE node
I20240502 13:59:36.782887 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 13:59:36.782932 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 13:59:36.782953 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658376782952 us; error 0 us; skew 500 ppm
I20240502 13:59:36.783779 581 webserver.cc:460] Webserver started at http://127.0.145.67:46517/ using document root <none> and password file <none>
I20240502 13:59:36.783915 581 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:36.783963 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:36.784031 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:36.784410 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658376531600-581-0/minicluster-data/ts-2-root/instance:
uuid: "011995a95dab429a9a0938c1639fc963"
format_stamp: "Formatted at 2024-05-02 13:59:36 on dist-test-slave-1z5j"
I20240502 13:59:36.786059 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 13:59:36.786993 996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:36.787247 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:36.787305 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658376531600-581-0/minicluster-data/ts-2-root
uuid: "011995a95dab429a9a0938c1639fc963"
format_stamp: "Formatted at 2024-05-02 13:59:36 on dist-test-slave-1z5j"
I20240502 13:59:36.787359 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658376531600-581-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658376531600-581-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658376531600-581-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:36.801365 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:36.801638 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:36.801853 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:36.802233 581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:36.802270 581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:36.802306 581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:36.802333 581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:36.806694 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.67:44213
I20240502 13:59:36.806830 1059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.67:44213 every 8 connection(s)
I20240502 13:59:36.811770 1060 heartbeater.cc:347] Connected to a master server at 127.0.145.124:41133
I20240502 13:59:36.811846 1060 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:36.812021 1060 heartbeater.cc:511] Master 127.0.145.124:41133 requested a full tablet report, sending...
I20240502 13:59:36.812616 744 ts_manager.cc:194] Registered new tserver with Master: 011995a95dab429a9a0938c1639fc963 (127.0.145.67:44213)
I20240502 13:59:36.813062 1062 heartbeater.cc:347] Connected to a master server at 127.0.145.125:46463
I20240502 13:59:36.813135 1062 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:36.813311 1061 heartbeater.cc:347] Connected to a master server at 127.0.145.126:39243
I20240502 13:59:36.813313 1062 heartbeater.cc:511] Master 127.0.145.125:46463 requested a full tablet report, sending...
I20240502 13:59:36.813424 1061 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:36.813578 1061 heartbeater.cc:511] Master 127.0.145.126:39243 requested a full tablet report, sending...
I20240502 13:59:36.813925 678 ts_manager.cc:194] Registered new tserver with Master: 011995a95dab429a9a0938c1639fc963 (127.0.145.67:44213)
I20240502 13:59:36.814460 611 ts_manager.cc:194] Registered new tserver with Master: 011995a95dab429a9a0938c1639fc963 (127.0.145.67:44213)
I20240502 13:59:36.815037 581 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007838519s
I20240502 13:59:36.835739 812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:36.835897 812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793b98f34d2e4038917c8b6d1e5e4a8b" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 39243 } } peers { permanent_uuid: "2d0dd09a40324e1c8cb7dc95b25f6375" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 46463 } } peers { permanent_uuid: "4008e3433a3941df8f20e1059c4ed97d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 41133 } }
I20240502 13:59:36.836617 812 leader_election.cc:288] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793b98f34d2e4038917c8b6d1e5e4a8b (127.0.145.126:39243), 4008e3433a3941df8f20e1059c4ed97d (127.0.145.124:41133)
I20240502 13:59:36.837311 754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d0dd09a40324e1c8cb7dc95b25f6375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4008e3433a3941df8f20e1059c4ed97d" is_pre_election: true
I20240502 13:59:36.837327 621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d0dd09a40324e1c8cb7dc95b25f6375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793b98f34d2e4038917c8b6d1e5e4a8b" is_pre_election: true
I20240502 13:59:36.837553 621 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 793b98f34d2e4038917c8b6d1e5e4a8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d0dd09a40324e1c8cb7dc95b25f6375 in term 0.
I20240502 13:59:36.837550 754 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4008e3433a3941df8f20e1059c4ed97d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d0dd09a40324e1c8cb7dc95b25f6375 in term 0.
I20240502 13:59:36.837944 658 leader_election.cc:302] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d0dd09a40324e1c8cb7dc95b25f6375, 4008e3433a3941df8f20e1059c4ed97d; no voters:
I20240502 13:59:36.838164 812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:36.838223 812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:36.838266 812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:36.839031 812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793b98f34d2e4038917c8b6d1e5e4a8b" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 39243 } } peers { permanent_uuid: "2d0dd09a40324e1c8cb7dc95b25f6375" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 46463 } } peers { permanent_uuid: "4008e3433a3941df8f20e1059c4ed97d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 41133 } }
I20240502 13:59:36.839293 812 leader_election.cc:288] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375 [CANDIDATE]: Term 1 election: Requested vote from peers 793b98f34d2e4038917c8b6d1e5e4a8b (127.0.145.126:39243), 4008e3433a3941df8f20e1059c4ed97d (127.0.145.124:41133)
I20240502 13:59:36.839557 621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d0dd09a40324e1c8cb7dc95b25f6375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793b98f34d2e4038917c8b6d1e5e4a8b"
I20240502 13:59:36.839608 754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d0dd09a40324e1c8cb7dc95b25f6375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4008e3433a3941df8f20e1059c4ed97d"
I20240502 13:59:36.839643 621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 793b98f34d2e4038917c8b6d1e5e4a8b [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:36.839725 754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4008e3433a3941df8f20e1059c4ed97d [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:36.840447 621 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 793b98f34d2e4038917c8b6d1e5e4a8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d0dd09a40324e1c8cb7dc95b25f6375 in term 1.
I20240502 13:59:36.840670 659 leader_election.cc:302] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d0dd09a40324e1c8cb7dc95b25f6375, 793b98f34d2e4038917c8b6d1e5e4a8b; no voters:
I20240502 13:59:36.840785 754 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4008e3433a3941df8f20e1059c4ed97d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d0dd09a40324e1c8cb7dc95b25f6375 in term 1.
I20240502 13:59:36.840834 812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:36.841028 812 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375 [term 1 LEADER]: Becoming Leader. State: Replica: 2d0dd09a40324e1c8cb7dc95b25f6375, State: Running, Role: LEADER
I20240502 13:59:36.841291 812 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793b98f34d2e4038917c8b6d1e5e4a8b" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 39243 } } peers { permanent_uuid: "2d0dd09a40324e1c8cb7dc95b25f6375" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 46463 } } peers { permanent_uuid: "4008e3433a3941df8f20e1059c4ed97d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 41133 } }
I20240502 13:59:36.842774 1069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d0dd09a40324e1c8cb7dc95b25f6375. Latest consensus state: current_term: 1 leader_uuid: "2d0dd09a40324e1c8cb7dc95b25f6375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793b98f34d2e4038917c8b6d1e5e4a8b" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 39243 } } peers { permanent_uuid: "2d0dd09a40324e1c8cb7dc95b25f6375" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 46463 } } peers { permanent_uuid: "4008e3433a3941df8f20e1059c4ed97d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 41133 } } }
I20240502 13:59:36.842871 1069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375 [sys.catalog]: This master's current role is: LEADER
I20240502 13:59:36.843174 1071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 13:59:36.843938 1071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 13:59:36.848599 754 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4008e3433a3941df8f20e1059c4ed97d [term 1 FOLLOWER]: Refusing update from remote peer 2d0dd09a40324e1c8cb7dc95b25f6375: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:36.848599 621 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 793b98f34d2e4038917c8b6d1e5e4a8b [term 1 FOLLOWER]: Refusing update from remote peer 2d0dd09a40324e1c8cb7dc95b25f6375: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:36.849087 1069 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4008e3433a3941df8f20e1059c4ed97d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:36.849249 812 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793b98f34d2e4038917c8b6d1e5e4a8b" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 39243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:36.850924 799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 793b98f34d2e4038917c8b6d1e5e4a8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d0dd09a40324e1c8cb7dc95b25f6375. Latest consensus state: current_term: 1 leader_uuid: "2d0dd09a40324e1c8cb7dc95b25f6375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793b98f34d2e4038917c8b6d1e5e4a8b" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 39243 } } peers { permanent_uuid: "2d0dd09a40324e1c8cb7dc95b25f6375" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 46463 } } peers { permanent_uuid: "4008e3433a3941df8f20e1059c4ed97d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 41133 } } }
I20240502 13:59:36.850993 785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4008e3433a3941df8f20e1059c4ed97d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d0dd09a40324e1c8cb7dc95b25f6375. Latest consensus state: current_term: 1 leader_uuid: "2d0dd09a40324e1c8cb7dc95b25f6375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793b98f34d2e4038917c8b6d1e5e4a8b" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 39243 } } peers { permanent_uuid: "2d0dd09a40324e1c8cb7dc95b25f6375" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 46463 } } peers { permanent_uuid: "4008e3433a3941df8f20e1059c4ed97d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 41133 } } }
I20240502 13:59:36.851119 785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4008e3433a3941df8f20e1059c4ed97d [sys.catalog]: This master's current role is: FOLLOWER
I20240502 13:59:36.851004 799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 793b98f34d2e4038917c8b6d1e5e4a8b [sys.catalog]: This master's current role is: FOLLOWER
I20240502 13:59:36.852207 812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d0dd09a40324e1c8cb7dc95b25f6375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793b98f34d2e4038917c8b6d1e5e4a8b" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 39243 } } peers { permanent_uuid: "2d0dd09a40324e1c8cb7dc95b25f6375" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 46463 } } peers { permanent_uuid: "4008e3433a3941df8f20e1059c4ed97d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 41133 } } }
I20240502 13:59:36.852298 812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375 [sys.catalog]: This master's current role is: LEADER
I20240502 13:59:36.852478 812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d0dd09a40324e1c8cb7dc95b25f6375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793b98f34d2e4038917c8b6d1e5e4a8b" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 39243 } } peers { permanent_uuid: "2d0dd09a40324e1c8cb7dc95b25f6375" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 46463 } } peers { permanent_uuid: "4008e3433a3941df8f20e1059c4ed97d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 41133 } } }
I20240502 13:59:36.852561 812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375 [sys.catalog]: This master's current role is: LEADER
I20240502 13:59:36.852723 785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4008e3433a3941df8f20e1059c4ed97d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d0dd09a40324e1c8cb7dc95b25f6375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793b98f34d2e4038917c8b6d1e5e4a8b" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 39243 } } peers { permanent_uuid: "2d0dd09a40324e1c8cb7dc95b25f6375" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 46463 } } peers { permanent_uuid: "4008e3433a3941df8f20e1059c4ed97d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 41133 } } }
I20240502 13:59:36.852843 785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4008e3433a3941df8f20e1059c4ed97d [sys.catalog]: This master's current role is: FOLLOWER
I20240502 13:59:36.853108 799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 793b98f34d2e4038917c8b6d1e5e4a8b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d0dd09a40324e1c8cb7dc95b25f6375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793b98f34d2e4038917c8b6d1e5e4a8b" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 39243 } } peers { permanent_uuid: "2d0dd09a40324e1c8cb7dc95b25f6375" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 46463 } } peers { permanent_uuid: "4008e3433a3941df8f20e1059c4ed97d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 41133 } } }
I20240502 13:59:36.853196 799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 793b98f34d2e4038917c8b6d1e5e4a8b [sys.catalog]: This master's current role is: FOLLOWER
I20240502 13:59:36.853832 1074 mvcc.cc:205] Tried to move back new op lower bound from 7023240711565824000 to 7023240711544188928. Current Snapshot: MvccSnapshot[applied={T|T < 7023240711565824000}]
I20240502 13:59:36.854413 1071 catalog_manager.cc:1347] Generated new cluster ID: 79d3ceb1653341cc9138594e7c69732d
I20240502 13:59:36.854468 1071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 13:59:36.874734 1071 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 13:59:36.875491 1071 catalog_manager.cc:1504] Loading token signing keys...
I20240502 13:59:36.883931 1071 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375: Generated new TSK 0
I20240502 13:59:36.885051 1071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 13:59:37.696630 796 catalog_manager.cc:1259] Loaded cluster ID: 79d3ceb1653341cc9138594e7c69732d
I20240502 13:59:37.696730 796 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4008e3433a3941df8f20e1059c4ed97d: loading cluster ID for follower catalog manager: success
I20240502 13:59:37.698048 796 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4008e3433a3941df8f20e1059c4ed97d: acquiring CA information for follower catalog manager: success
I20240502 13:59:37.698626 810 catalog_manager.cc:1259] Loaded cluster ID: 79d3ceb1653341cc9138594e7c69732d
I20240502 13:59:37.698679 810 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 793b98f34d2e4038917c8b6d1e5e4a8b: loading cluster ID for follower catalog manager: success
I20240502 13:59:37.698681 796 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4008e3433a3941df8f20e1059c4ed97d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 13:59:37.699704 810 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 793b98f34d2e4038917c8b6d1e5e4a8b: acquiring CA information for follower catalog manager: success
I20240502 13:59:37.700161 810 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 793b98f34d2e4038917c8b6d1e5e4a8b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 13:59:37.757649 678 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36266
I20240502 13:59:37.783664 678 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36274
I20240502 13:59:37.815186 678 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36276
I20240502 13:59:38.712328 823 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 13:59:38.758791 902 heartbeater.cc:503] Master 127.0.145.125:46463 was elected leader, sending a full tablet report...
I20240502 13:59:38.784353 982 heartbeater.cc:503] Master 127.0.145.125:46463 was elected leader, sending a full tablet report...
I20240502 13:59:38.815894 1062 heartbeater.cc:503] Master 127.0.145.125:46463 was elected leader, sending a full tablet report...
I20240502 13:59:38.967180 581 test_util.cc:268] Using random seed: 60221574
I20240502 13:59:38.983788 678 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36290:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 13:59:39.000661 944 tablet_service.cc:1467] Processing CreateTablet for tablet fd59d6ebc1ad4c399ef246421100e74e (DEFAULT_TABLE table=test-workload [id=1c9a4f44355d42a39dbc293ff59741ab]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 13:59:39.000814 861 tablet_service.cc:1467] Processing CreateTablet for tablet 34ef2a5696eb46d9a72a236ff50bc6fb (DEFAULT_TABLE table=test-workload [id=1c9a4f44355d42a39dbc293ff59741ab]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 13:59:39.001125 944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd59d6ebc1ad4c399ef246421100e74e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:39.001142 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 8b8f0588724b4271a6496241da5d96a2 (DEFAULT_TABLE table=test-workload [id=1c9a4f44355d42a39dbc293ff59741ab]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 13:59:39.001331 860 tablet_service.cc:1467] Processing CreateTablet for tablet 83d63b5e0acb45bc9c1031c6eb194911 (DEFAULT_TABLE table=test-workload [id=1c9a4f44355d42a39dbc293ff59741ab]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 13:59:39.001425 1024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b8f0588724b4271a6496241da5d96a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:39.001125 861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34ef2a5696eb46d9a72a236ff50bc6fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:39.001659 860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83d63b5e0acb45bc9c1031c6eb194911. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:39.003892 1110 tablet_bootstrap.cc:492] T 34ef2a5696eb46d9a72a236ff50bc6fb P 46b2b951921d44d1a8323c7a250e6c7d: Bootstrap starting.
I20240502 13:59:39.004621 1110 tablet_bootstrap.cc:654] T 34ef2a5696eb46d9a72a236ff50bc6fb P 46b2b951921d44d1a8323c7a250e6c7d: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:39.004931 1108 tablet_bootstrap.cc:492] T 8b8f0588724b4271a6496241da5d96a2 P 011995a95dab429a9a0938c1639fc963: Bootstrap starting.
I20240502 13:59:39.005601 1110 tablet_bootstrap.cc:492] T 34ef2a5696eb46d9a72a236ff50bc6fb P 46b2b951921d44d1a8323c7a250e6c7d: No bootstrap required, opened a new log
I20240502 13:59:39.005641 1108 tablet_bootstrap.cc:654] T 8b8f0588724b4271a6496241da5d96a2 P 011995a95dab429a9a0938c1639fc963: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:39.005708 1110 ts_tablet_manager.cc:1367] T 34ef2a5696eb46d9a72a236ff50bc6fb P 46b2b951921d44d1a8323c7a250e6c7d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 13:59:39.006112 1110 raft_consensus.cc:348] T 34ef2a5696eb46d9a72a236ff50bc6fb P 46b2b951921d44d1a8323c7a250e6c7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b2b951921d44d1a8323c7a250e6c7d" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 39525 } }
I20240502 13:59:39.006199 1109 tablet_bootstrap.cc:492] T fd59d6ebc1ad4c399ef246421100e74e P 0a4408b75e284c6580624cd2a730d82f: Bootstrap starting.
I20240502 13:59:39.006219 1110 raft_consensus.cc:374] T 34ef2a5696eb46d9a72a236ff50bc6fb P 46b2b951921d44d1a8323c7a250e6c7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:39.006249 1110 raft_consensus.cc:729] T 34ef2a5696eb46d9a72a236ff50bc6fb P 46b2b951921d44d1a8323c7a250e6c7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46b2b951921d44d1a8323c7a250e6c7d, State: Initialized, Role: FOLLOWER
I20240502 13:59:39.006421 1110 consensus_queue.cc:260] T 34ef2a5696eb46d9a72a236ff50bc6fb P 46b2b951921d44d1a8323c7a250e6c7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b2b951921d44d1a8323c7a250e6c7d" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 39525 } }
I20240502 13:59:39.006486 1108 tablet_bootstrap.cc:492] T 8b8f0588724b4271a6496241da5d96a2 P 011995a95dab429a9a0938c1639fc963: No bootstrap required, opened a new log
I20240502 13:59:39.006520 1110 raft_consensus.cc:388] T 34ef2a5696eb46d9a72a236ff50bc6fb P 46b2b951921d44d1a8323c7a250e6c7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:39.006554 1108 ts_tablet_manager.cc:1367] T 8b8f0588724b4271a6496241da5d96a2 P 011995a95dab429a9a0938c1639fc963: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:39.006605 1110 raft_consensus.cc:482] T 34ef2a5696eb46d9a72a236ff50bc6fb P 46b2b951921d44d1a8323c7a250e6c7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:39.006649 1110 raft_consensus.cc:3037] T 34ef2a5696eb46d9a72a236ff50bc6fb P 46b2b951921d44d1a8323c7a250e6c7d [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:39.006911 1108 raft_consensus.cc:348] T 8b8f0588724b4271a6496241da5d96a2 P 011995a95dab429a9a0938c1639fc963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011995a95dab429a9a0938c1639fc963" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 44213 } }
I20240502 13:59:39.006973 1109 tablet_bootstrap.cc:654] T fd59d6ebc1ad4c399ef246421100e74e P 0a4408b75e284c6580624cd2a730d82f: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:39.006984 1108 raft_consensus.cc:374] T 8b8f0588724b4271a6496241da5d96a2 P 011995a95dab429a9a0938c1639fc963 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:39.007059 1108 raft_consensus.cc:729] T 8b8f0588724b4271a6496241da5d96a2 P 011995a95dab429a9a0938c1639fc963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 011995a95dab429a9a0938c1639fc963, State: Initialized, Role: FOLLOWER
I20240502 13:59:39.007196 1108 consensus_queue.cc:260] T 8b8f0588724b4271a6496241da5d96a2 P 011995a95dab429a9a0938c1639fc963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011995a95dab429a9a0938c1639fc963" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 44213 } }
I20240502 13:59:39.007265 1108 raft_consensus.cc:388] T 8b8f0588724b4271a6496241da5d96a2 P 011995a95dab429a9a0938c1639fc963 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:39.007315 1108 raft_consensus.cc:482] T 8b8f0588724b4271a6496241da5d96a2 P 011995a95dab429a9a0938c1639fc963 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:39.007364 1108 raft_consensus.cc:3037] T 8b8f0588724b4271a6496241da5d96a2 P 011995a95dab429a9a0938c1639fc963 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:39.007433 1110 raft_consensus.cc:504] T 34ef2a5696eb46d9a72a236ff50bc6fb P 46b2b951921d44d1a8323c7a250e6c7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b2b951921d44d1a8323c7a250e6c7d" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 39525 } }
I20240502 13:59:39.007611 1110 leader_election.cc:302] T 34ef2a5696eb46d9a72a236ff50bc6fb P 46b2b951921d44d1a8323c7a250e6c7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46b2b951921d44d1a8323c7a250e6c7d; no voters:
I20240502 13:59:39.007773 1110 leader_election.cc:288] T 34ef2a5696eb46d9a72a236ff50bc6fb P 46b2b951921d44d1a8323c7a250e6c7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:39.007854 1114 raft_consensus.cc:2781] T 34ef2a5696eb46d9a72a236ff50bc6fb P 46b2b951921d44d1a8323c7a250e6c7d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:39.008059 1110 ts_tablet_manager.cc:1398] T 34ef2a5696eb46d9a72a236ff50bc6fb P 46b2b951921d44d1a8323c7a250e6c7d: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 13:59:39.008070 1114 raft_consensus.cc:686] T 34ef2a5696eb46d9a72a236ff50bc6fb P 46b2b951921d44d1a8323c7a250e6c7d [term 1 LEADER]: Becoming Leader. State: Replica: 46b2b951921d44d1a8323c7a250e6c7d, State: Running, Role: LEADER
I20240502 13:59:39.008301 1114 consensus_queue.cc:237] T 34ef2a5696eb46d9a72a236ff50bc6fb P 46b2b951921d44d1a8323c7a250e6c7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b2b951921d44d1a8323c7a250e6c7d" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 39525 } }
I20240502 13:59:39.008380 1108 raft_consensus.cc:504] T 8b8f0588724b4271a6496241da5d96a2 P 011995a95dab429a9a0938c1639fc963 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011995a95dab429a9a0938c1639fc963" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 44213 } }
I20240502 13:59:39.008524 1108 leader_election.cc:302] T 8b8f0588724b4271a6496241da5d96a2 P 011995a95dab429a9a0938c1639fc963 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 011995a95dab429a9a0938c1639fc963; no voters:
I20240502 13:59:39.008720 1108 leader_election.cc:288] T 8b8f0588724b4271a6496241da5d96a2 P 011995a95dab429a9a0938c1639fc963 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:39.008806 1118 raft_consensus.cc:2781] T 8b8f0588724b4271a6496241da5d96a2 P 011995a95dab429a9a0938c1639fc963 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:39.009008 1118 raft_consensus.cc:686] T 8b8f0588724b4271a6496241da5d96a2 P 011995a95dab429a9a0938c1639fc963 [term 1 LEADER]: Becoming Leader. State: Replica: 011995a95dab429a9a0938c1639fc963, State: Running, Role: LEADER
I20240502 13:59:39.009008 1108 ts_tablet_manager.cc:1398] T 8b8f0588724b4271a6496241da5d96a2 P 011995a95dab429a9a0938c1639fc963: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:39.009172 1118 consensus_queue.cc:237] T 8b8f0588724b4271a6496241da5d96a2 P 011995a95dab429a9a0938c1639fc963 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011995a95dab429a9a0938c1639fc963" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 44213 } }
I20240502 13:59:39.010915 677 catalog_manager.cc:5484] T 8b8f0588724b4271a6496241da5d96a2 P 011995a95dab429a9a0938c1639fc963 reported cstate change: term changed from 0 to 1, leader changed from <none> to 011995a95dab429a9a0938c1639fc963 (127.0.145.67). New cstate: current_term: 1 leader_uuid: "011995a95dab429a9a0938c1639fc963" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011995a95dab429a9a0938c1639fc963" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 44213 } health_report { overall_health: HEALTHY } } }
I20240502 13:59:39.010821 678 catalog_manager.cc:5484] T 34ef2a5696eb46d9a72a236ff50bc6fb P 46b2b951921d44d1a8323c7a250e6c7d reported cstate change: term changed from 0 to 1, leader changed from <none> to 46b2b951921d44d1a8323c7a250e6c7d (127.0.145.65). New cstate: current_term: 1 leader_uuid: "46b2b951921d44d1a8323c7a250e6c7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b2b951921d44d1a8323c7a250e6c7d" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 39525 } health_report { overall_health: HEALTHY } } }
I20240502 13:59:39.013909 1110 tablet_bootstrap.cc:492] T 83d63b5e0acb45bc9c1031c6eb194911 P 46b2b951921d44d1a8323c7a250e6c7d: Bootstrap starting.
I20240502 13:59:39.015188 1110 tablet_bootstrap.cc:654] T 83d63b5e0acb45bc9c1031c6eb194911 P 46b2b951921d44d1a8323c7a250e6c7d: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:39.016249 1109 tablet_bootstrap.cc:492] T fd59d6ebc1ad4c399ef246421100e74e P 0a4408b75e284c6580624cd2a730d82f: No bootstrap required, opened a new log
I20240502 13:59:39.016340 1109 ts_tablet_manager.cc:1367] T fd59d6ebc1ad4c399ef246421100e74e P 0a4408b75e284c6580624cd2a730d82f: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240502 13:59:39.016660 1109 raft_consensus.cc:348] T fd59d6ebc1ad4c399ef246421100e74e P 0a4408b75e284c6580624cd2a730d82f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a4408b75e284c6580624cd2a730d82f" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 34109 } }
I20240502 13:59:39.016732 1109 raft_consensus.cc:374] T fd59d6ebc1ad4c399ef246421100e74e P 0a4408b75e284c6580624cd2a730d82f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:39.016753 1109 raft_consensus.cc:729] T fd59d6ebc1ad4c399ef246421100e74e P 0a4408b75e284c6580624cd2a730d82f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a4408b75e284c6580624cd2a730d82f, State: Initialized, Role: FOLLOWER
I20240502 13:59:39.016889 1109 consensus_queue.cc:260] T fd59d6ebc1ad4c399ef246421100e74e P 0a4408b75e284c6580624cd2a730d82f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a4408b75e284c6580624cd2a730d82f" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 34109 } }
I20240502 13:59:39.016957 1109 raft_consensus.cc:388] T fd59d6ebc1ad4c399ef246421100e74e P 0a4408b75e284c6580624cd2a730d82f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:39.016989 1109 raft_consensus.cc:482] T fd59d6ebc1ad4c399ef246421100e74e P 0a4408b75e284c6580624cd2a730d82f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:39.017033 1109 raft_consensus.cc:3037] T fd59d6ebc1ad4c399ef246421100e74e P 0a4408b75e284c6580624cd2a730d82f [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:39.017632 1109 raft_consensus.cc:504] T fd59d6ebc1ad4c399ef246421100e74e P 0a4408b75e284c6580624cd2a730d82f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a4408b75e284c6580624cd2a730d82f" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 34109 } }
I20240502 13:59:39.017731 1109 leader_election.cc:302] T fd59d6ebc1ad4c399ef246421100e74e P 0a4408b75e284c6580624cd2a730d82f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a4408b75e284c6580624cd2a730d82f; no voters:
I20240502 13:59:39.017889 1109 leader_election.cc:288] T fd59d6ebc1ad4c399ef246421100e74e P 0a4408b75e284c6580624cd2a730d82f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:39.018180 1109 ts_tablet_manager.cc:1398] T fd59d6ebc1ad4c399ef246421100e74e P 0a4408b75e284c6580624cd2a730d82f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 13:59:39.020037 1130 raft_consensus.cc:2781] T fd59d6ebc1ad4c399ef246421100e74e P 0a4408b75e284c6580624cd2a730d82f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:39.020130 1130 raft_consensus.cc:686] T fd59d6ebc1ad4c399ef246421100e74e P 0a4408b75e284c6580624cd2a730d82f [term 1 LEADER]: Becoming Leader. State: Replica: 0a4408b75e284c6580624cd2a730d82f, State: Running, Role: LEADER
I20240502 13:59:39.020232 1130 consensus_queue.cc:237] T fd59d6ebc1ad4c399ef246421100e74e P 0a4408b75e284c6580624cd2a730d82f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a4408b75e284c6580624cd2a730d82f" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 34109 } }
I20240502 13:59:39.022006 676 catalog_manager.cc:5484] T fd59d6ebc1ad4c399ef246421100e74e P 0a4408b75e284c6580624cd2a730d82f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a4408b75e284c6580624cd2a730d82f (127.0.145.66). New cstate: current_term: 1 leader_uuid: "0a4408b75e284c6580624cd2a730d82f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a4408b75e284c6580624cd2a730d82f" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 34109 } health_report { overall_health: HEALTHY } } }
I20240502 13:59:39.041004 1110 tablet_bootstrap.cc:492] T 83d63b5e0acb45bc9c1031c6eb194911 P 46b2b951921d44d1a8323c7a250e6c7d: No bootstrap required, opened a new log
I20240502 13:59:39.041091 1110 ts_tablet_manager.cc:1367] T 83d63b5e0acb45bc9c1031c6eb194911 P 46b2b951921d44d1a8323c7a250e6c7d: Time spent bootstrapping tablet: real 0.033s user 0.000s sys 0.001s
I20240502 13:59:39.041410 1110 raft_consensus.cc:348] T 83d63b5e0acb45bc9c1031c6eb194911 P 46b2b951921d44d1a8323c7a250e6c7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b2b951921d44d1a8323c7a250e6c7d" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 39525 } }
I20240502 13:59:39.041496 1110 raft_consensus.cc:374] T 83d63b5e0acb45bc9c1031c6eb194911 P 46b2b951921d44d1a8323c7a250e6c7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:39.041525 1110 raft_consensus.cc:729] T 83d63b5e0acb45bc9c1031c6eb194911 P 46b2b951921d44d1a8323c7a250e6c7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46b2b951921d44d1a8323c7a250e6c7d, State: Initialized, Role: FOLLOWER
I20240502 13:59:39.041631 1110 consensus_queue.cc:260] T 83d63b5e0acb45bc9c1031c6eb194911 P 46b2b951921d44d1a8323c7a250e6c7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b2b951921d44d1a8323c7a250e6c7d" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 39525 } }
I20240502 13:59:39.041693 1110 raft_consensus.cc:388] T 83d63b5e0acb45bc9c1031c6eb194911 P 46b2b951921d44d1a8323c7a250e6c7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:39.041730 1110 raft_consensus.cc:482] T 83d63b5e0acb45bc9c1031c6eb194911 P 46b2b951921d44d1a8323c7a250e6c7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:39.041776 1110 raft_consensus.cc:3037] T 83d63b5e0acb45bc9c1031c6eb194911 P 46b2b951921d44d1a8323c7a250e6c7d [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:39.042384 1110 raft_consensus.cc:504] T 83d63b5e0acb45bc9c1031c6eb194911 P 46b2b951921d44d1a8323c7a250e6c7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b2b951921d44d1a8323c7a250e6c7d" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 39525 } }
I20240502 13:59:39.042496 1110 leader_election.cc:302] T 83d63b5e0acb45bc9c1031c6eb194911 P 46b2b951921d44d1a8323c7a250e6c7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46b2b951921d44d1a8323c7a250e6c7d; no voters:
I20240502 13:59:39.042584 1110 leader_election.cc:288] T 83d63b5e0acb45bc9c1031c6eb194911 P 46b2b951921d44d1a8323c7a250e6c7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:39.042639 1115 raft_consensus.cc:2781] T 83d63b5e0acb45bc9c1031c6eb194911 P 46b2b951921d44d1a8323c7a250e6c7d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:39.042768 1110 ts_tablet_manager.cc:1398] T 83d63b5e0acb45bc9c1031c6eb194911 P 46b2b951921d44d1a8323c7a250e6c7d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 13:59:39.042788 1115 raft_consensus.cc:686] T 83d63b5e0acb45bc9c1031c6eb194911 P 46b2b951921d44d1a8323c7a250e6c7d [term 1 LEADER]: Becoming Leader. State: Replica: 46b2b951921d44d1a8323c7a250e6c7d, State: Running, Role: LEADER
I20240502 13:59:39.042894 1115 consensus_queue.cc:237] T 83d63b5e0acb45bc9c1031c6eb194911 P 46b2b951921d44d1a8323c7a250e6c7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b2b951921d44d1a8323c7a250e6c7d" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 39525 } }
I20240502 13:59:39.043999 676 catalog_manager.cc:5484] T 83d63b5e0acb45bc9c1031c6eb194911 P 46b2b951921d44d1a8323c7a250e6c7d reported cstate change: term changed from 0 to 1, leader changed from <none> to 46b2b951921d44d1a8323c7a250e6c7d (127.0.145.65). New cstate: current_term: 1 leader_uuid: "46b2b951921d44d1a8323c7a250e6c7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b2b951921d44d1a8323c7a250e6c7d" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 39525 } health_report { overall_health: HEALTHY } } }
I20240502 13:59:39.056943 581 tablet_server.cc:178] TabletServer@127.0.145.65:0 shutting down...
I20240502 13:59:39.063082 581 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 13:59:39.063305 581 tablet_replica.cc:330] T 83d63b5e0acb45bc9c1031c6eb194911 P 46b2b951921d44d1a8323c7a250e6c7d: stopping tablet replica
I20240502 13:59:39.063513 581 raft_consensus.cc:2229] T 83d63b5e0acb45bc9c1031c6eb194911 P 46b2b951921d44d1a8323c7a250e6c7d [term 1 LEADER]: Raft consensus shutting down.
I20240502 13:59:39.063617 581 raft_consensus.cc:2258] T 83d63b5e0acb45bc9c1031c6eb194911 P 46b2b951921d44d1a8323c7a250e6c7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:39.064496 581 tablet_replica.cc:330] T 34ef2a5696eb46d9a72a236ff50bc6fb P 46b2b951921d44d1a8323c7a250e6c7d: stopping tablet replica
I20240502 13:59:39.064565 581 raft_consensus.cc:2229] T 34ef2a5696eb46d9a72a236ff50bc6fb P 46b2b951921d44d1a8323c7a250e6c7d [term 1 LEADER]: Raft consensus shutting down.
I20240502 13:59:39.064633 581 raft_consensus.cc:2258] T 34ef2a5696eb46d9a72a236ff50bc6fb P 46b2b951921d44d1a8323c7a250e6c7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:39.077889 581 tablet_server.cc:194] TabletServer@127.0.145.65:0 shutdown complete.
I20240502 13:59:39.081104 581 tablet_server.cc:178] TabletServer@127.0.145.66:0 shutting down...
I20240502 13:59:39.084393 581 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 13:59:39.084579 581 tablet_replica.cc:330] T fd59d6ebc1ad4c399ef246421100e74e P 0a4408b75e284c6580624cd2a730d82f: stopping tablet replica
I20240502 13:59:39.084666 581 raft_consensus.cc:2229] T fd59d6ebc1ad4c399ef246421100e74e P 0a4408b75e284c6580624cd2a730d82f [term 1 LEADER]: Raft consensus shutting down.
I20240502 13:59:39.084738 581 raft_consensus.cc:2258] T fd59d6ebc1ad4c399ef246421100e74e P 0a4408b75e284c6580624cd2a730d82f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:39.096953 581 tablet_server.cc:194] TabletServer@127.0.145.66:0 shutdown complete.
I20240502 13:59:39.098353 581 tablet_server.cc:178] TabletServer@127.0.145.67:0 shutting down...
I20240502 13:59:39.101447 581 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 13:59:39.101629 581 tablet_replica.cc:330] T 8b8f0588724b4271a6496241da5d96a2 P 011995a95dab429a9a0938c1639fc963: stopping tablet replica
I20240502 13:59:39.101719 581 raft_consensus.cc:2229] T 8b8f0588724b4271a6496241da5d96a2 P 011995a95dab429a9a0938c1639fc963 [term 1 LEADER]: Raft consensus shutting down.
I20240502 13:59:39.101790 581 raft_consensus.cc:2258] T 8b8f0588724b4271a6496241da5d96a2 P 011995a95dab429a9a0938c1639fc963 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:39.113997 581 tablet_server.cc:194] TabletServer@127.0.145.67:0 shutdown complete.
I20240502 13:59:39.115339 581 master.cc:537] Master@127.0.145.126:39243 shutting down...
I20240502 13:59:39.117851 581 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 793b98f34d2e4038917c8b6d1e5e4a8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 13:59:39.117944 581 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 793b98f34d2e4038917c8b6d1e5e4a8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:39.118008 581 tablet_replica.cc:330] T 00000000000000000000000000000000 P 793b98f34d2e4038917c8b6d1e5e4a8b: stopping tablet replica
I20240502 13:59:39.137187 581 master.cc:552] Master@127.0.145.126:39243 shutdown complete.
I20240502 13:59:39.139364 581 master.cc:537] Master@127.0.145.125:46463 shutting down...
I20240502 13:59:39.141824 581 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375 [term 1 LEADER]: Raft consensus shutting down.
I20240502 13:59:39.142036 581 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:39.142094 581 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2d0dd09a40324e1c8cb7dc95b25f6375: stopping tablet replica
I20240502 13:59:39.154664 581 master.cc:552] Master@127.0.145.125:46463 shutdown complete.
I20240502 13:59:39.157027 581 master.cc:537] Master@127.0.145.124:41133 shutting down...
I20240502 13:59:39.159636 581 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4008e3433a3941df8f20e1059c4ed97d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 13:59:39.159739 581 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4008e3433a3941df8f20e1059c4ed97d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:39.159791 581 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4008e3433a3941df8f20e1059c4ed97d: stopping tablet replica
I20240502 13:59:39.161921 581 master.cc:552] Master@127.0.145.124:41133 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (2627 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20240502 13:59:39.167932 581 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.145.126:41703
I20240502 13:59:39.168198 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:39.169514 1135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:39.170015 1138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:39.170374 1136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:39.170727 581 server_base.cc:1034] running on GCE node
I20240502 13:59:39.170835 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 13:59:39.170866 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 13:59:39.170886 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658379170885 us; error 0 us; skew 500 ppm
I20240502 13:59:39.171633 581 webserver.cc:460] Webserver started at http://127.0.145.126:43777/ using document root <none> and password file <none>
I20240502 13:59:39.171748 581 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:39.171792 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:39.171852 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:39.172188 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658376531600-581-0/minicluster-data/master-0-root/instance:
uuid: "2c01d5eee794439b85e4f9d59d986b98"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-1z5j"
I20240502 13:59:39.173458 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 13:59:39.174211 1143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:39.174409 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:39.174469 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658376531600-581-0/minicluster-data/master-0-root
uuid: "2c01d5eee794439b85e4f9d59d986b98"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-1z5j"
I20240502 13:59:39.174528 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658376531600-581-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658376531600-581-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658376531600-581-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:39.189743 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:39.190057 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:39.193394 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.126:41703
I20240502 13:59:39.195674 1196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.126:41703 every 8 connection(s)
I20240502 13:59:39.196036 1197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:39.197572 1197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c01d5eee794439b85e4f9d59d986b98: Bootstrap starting.
I20240502 13:59:39.198206 1197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c01d5eee794439b85e4f9d59d986b98: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:39.199002 1197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c01d5eee794439b85e4f9d59d986b98: No bootstrap required, opened a new log
I20240502 13:59:39.199292 1197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c01d5eee794439b85e4f9d59d986b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c01d5eee794439b85e4f9d59d986b98" member_type: VOTER }
I20240502 13:59:39.199366 1197 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c01d5eee794439b85e4f9d59d986b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:39.199396 1197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c01d5eee794439b85e4f9d59d986b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c01d5eee794439b85e4f9d59d986b98, State: Initialized, Role: FOLLOWER
I20240502 13:59:39.199478 1197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c01d5eee794439b85e4f9d59d986b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c01d5eee794439b85e4f9d59d986b98" member_type: VOTER }
I20240502 13:59:39.199528 1197 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c01d5eee794439b85e4f9d59d986b98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:39.199556 1197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c01d5eee794439b85e4f9d59d986b98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:39.199587 1197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c01d5eee794439b85e4f9d59d986b98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:39.200143 1197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c01d5eee794439b85e4f9d59d986b98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c01d5eee794439b85e4f9d59d986b98" member_type: VOTER }
I20240502 13:59:39.200241 1197 leader_election.cc:302] T 00000000000000000000000000000000 P 2c01d5eee794439b85e4f9d59d986b98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c01d5eee794439b85e4f9d59d986b98; no voters:
I20240502 13:59:39.200369 1197 leader_election.cc:288] T 00000000000000000000000000000000 P 2c01d5eee794439b85e4f9d59d986b98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:39.200464 1200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c01d5eee794439b85e4f9d59d986b98 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:39.200631 1200 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c01d5eee794439b85e4f9d59d986b98 [term 1 LEADER]: Becoming Leader. State: Replica: 2c01d5eee794439b85e4f9d59d986b98, State: Running, Role: LEADER
I20240502 13:59:39.200727 1197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c01d5eee794439b85e4f9d59d986b98 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 13:59:39.200739 1200 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c01d5eee794439b85e4f9d59d986b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c01d5eee794439b85e4f9d59d986b98" member_type: VOTER }
I20240502 13:59:39.201184 1202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c01d5eee794439b85e4f9d59d986b98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c01d5eee794439b85e4f9d59d986b98. Latest consensus state: current_term: 1 leader_uuid: "2c01d5eee794439b85e4f9d59d986b98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c01d5eee794439b85e4f9d59d986b98" member_type: VOTER } }
I20240502 13:59:39.201247 1202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c01d5eee794439b85e4f9d59d986b98 [sys.catalog]: This master's current role is: LEADER
I20240502 13:59:39.201159 1201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c01d5eee794439b85e4f9d59d986b98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c01d5eee794439b85e4f9d59d986b98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c01d5eee794439b85e4f9d59d986b98" member_type: VOTER } }
I20240502 13:59:39.201388 1201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c01d5eee794439b85e4f9d59d986b98 [sys.catalog]: This master's current role is: LEADER
I20240502 13:59:39.201519 1204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 13:59:39.202172 1204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 13:59:39.202481 581 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 13:59:39.203720 1204 catalog_manager.cc:1347] Generated new cluster ID: a2ce2d82c0dc4e28b7462d4f46f3ca0e
I20240502 13:59:39.203776 1204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 13:59:39.229480 1204 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 13:59:39.230003 1204 catalog_manager.cc:1504] Loading token signing keys...
I20240502 13:59:39.243301 1204 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2c01d5eee794439b85e4f9d59d986b98: Generated new TSK 0
I20240502 13:59:39.243460 1204 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 13:59:39.266713 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:39.268343 1218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:39.268811 1221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:39.269192 1219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:39.269549 581 server_base.cc:1034] running on GCE node
I20240502 13:59:39.269683 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 13:59:39.269719 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 13:59:39.269738 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658379269738 us; error 0 us; skew 500 ppm
I20240502 13:59:39.270522 581 webserver.cc:460] Webserver started at http://127.0.145.65:36665/ using document root <none> and password file <none>
I20240502 13:59:39.270651 581 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:39.270695 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:39.270763 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:39.271111 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658376531600-581-0/minicluster-data/ts-0-root/instance:
uuid: "fb3125d57f2741698f075887542fc1fc"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-1z5j"
I20240502 13:59:39.272364 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:39.273185 1226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:39.273384 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:39.273445 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658376531600-581-0/minicluster-data/ts-0-root
uuid: "fb3125d57f2741698f075887542fc1fc"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-1z5j"
I20240502 13:59:39.273507 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658376531600-581-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658376531600-581-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658376531600-581-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:39.292016 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:39.292274 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:39.292491 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:39.292881 581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:39.292917 581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:39.292960 581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:39.292987 581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:39.296485 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.65:34005
I20240502 13:59:39.296526 1289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.65:34005 every 8 connection(s)
I20240502 13:59:39.297168 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:39.298863 1295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:39.298964 1294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:39.299335 581 server_base.cc:1034] running on GCE node
W20240502 13:59:39.299605 1297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:39.299814 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 13:59:39.299865 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 13:59:39.299885 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658379299885 us; error 0 us; skew 500 ppm
I20240502 13:59:39.300685 581 webserver.cc:460] Webserver started at http://127.0.145.66:39133/ using document root <none> and password file <none>
I20240502 13:59:39.300837 581 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:39.300887 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:39.300944 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:39.301093 1290 heartbeater.cc:347] Connected to a master server at 127.0.145.126:41703
I20240502 13:59:39.301177 1290 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:39.301304 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658376531600-581-0/minicluster-data/ts-1-root/instance:
uuid: "8feeac9977814b8b901607ec4071efdf"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-1z5j"
I20240502 13:59:39.301403 1290 heartbeater.cc:511] Master 127.0.145.126:41703 requested a full tablet report, sending...
I20240502 13:59:39.301947 1161 ts_manager.cc:194] Registered new tserver with Master: fb3125d57f2741698f075887542fc1fc (127.0.145.65:34005)
I20240502 13:59:39.302722 1161 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41606
I20240502 13:59:39.302728 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240502 13:59:39.303599 1302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:39.303817 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:39.303875 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658376531600-581-0/minicluster-data/ts-1-root
uuid: "8feeac9977814b8b901607ec4071efdf"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-1z5j"
I20240502 13:59:39.303932 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658376531600-581-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658376531600-581-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658376531600-581-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:39.314762 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:39.315009 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:39.315218 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:39.315577 581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:39.315611 581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:39.315645 581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:39.315673 581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:39.319132 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.66:45147
I20240502 13:59:39.319172 1367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.66:45147 every 8 connection(s)
I20240502 13:59:39.319728 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:39.321126 1371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:39.321449 1372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:39.321844 581 server_base.cc:1034] running on GCE node
W20240502 13:59:39.322124 1374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:39.322311 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 13:59:39.322358 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 13:59:39.322379 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658379322379 us; error 0 us; skew 500 ppm
I20240502 13:59:39.323267 1368 heartbeater.cc:347] Connected to a master server at 127.0.145.126:41703
I20240502 13:59:39.323277 581 webserver.cc:460] Webserver started at http://127.0.145.67:42063/ using document root <none> and password file <none>
I20240502 13:59:39.323366 1368 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:39.323432 581 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:39.323482 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:39.323546 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:39.323556 1368 heartbeater.cc:511] Master 127.0.145.126:41703 requested a full tablet report, sending...
I20240502 13:59:39.323879 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658376531600-581-0/minicluster-data/ts-2-root/instance:
uuid: "0c0b5fc6c9894f7ca9a083e986452dff"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-1z5j"
I20240502 13:59:39.324085 1161 ts_manager.cc:194] Registered new tserver with Master: 8feeac9977814b8b901607ec4071efdf (127.0.145.66:45147)
I20240502 13:59:39.324680 1161 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41614
I20240502 13:59:39.325452 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 13:59:39.326193 1379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:39.326386 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:39.326452 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658376531600-581-0/minicluster-data/ts-2-root
uuid: "0c0b5fc6c9894f7ca9a083e986452dff"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-1z5j"
I20240502 13:59:39.326510 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658376531600-581-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658376531600-581-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658376531600-581-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:39.339278 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:39.339569 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:39.339818 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:39.340166 581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:39.340201 581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:39.340236 581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:39.340269 581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:39.344236 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.67:41377
I20240502 13:59:39.344277 1443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.67:41377 every 8 connection(s)
I20240502 13:59:39.347801 1444 heartbeater.cc:347] Connected to a master server at 127.0.145.126:41703
I20240502 13:59:39.347869 1444 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:39.348057 1444 heartbeater.cc:511] Master 127.0.145.126:41703 requested a full tablet report, sending...
I20240502 13:59:39.348543 1161 ts_manager.cc:194] Registered new tserver with Master: 0c0b5fc6c9894f7ca9a083e986452dff (127.0.145.67:41377)
I20240502 13:59:39.349016 1161 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41630
I20240502 13:59:39.349046 581 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004473955s
I20240502 13:59:40.303545 1290 heartbeater.cc:503] Master 127.0.145.126:41703 was elected leader, sending a full tablet report...
I20240502 13:59:40.325485 1368 heartbeater.cc:503] Master 127.0.145.126:41703 was elected leader, sending a full tablet report...
I20240502 13:59:40.349769 1444 heartbeater.cc:503] Master 127.0.145.126:41703 was elected leader, sending a full tablet report...
I20240502 13:59:40.374210 581 test_util.cc:268] Using random seed: 61628608
I20240502 13:59:40.381250 1161 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41644:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 13:59:40.381772 1161 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 13:59:40.392832 1329 tablet_service.cc:1467] Processing CreateTablet for tablet 7a8684e4da8f46b6acace6ccac46991f (DEFAULT_TABLE table=test-workload [id=d2f0f25ba75b4b18b99f7ede77394cd1]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 13:59:40.393003 1330 tablet_service.cc:1467] Processing CreateTablet for tablet 2a57d4e298464ad580027819241d4735 (DEFAULT_TABLE table=test-workload [id=d2f0f25ba75b4b18b99f7ede77394cd1]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 13:59:40.393153 1408 tablet_service.cc:1467] Processing CreateTablet for tablet 2a57d4e298464ad580027819241d4735 (DEFAULT_TABLE table=test-workload [id=d2f0f25ba75b4b18b99f7ede77394cd1]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 13:59:40.393196 1326 tablet_service.cc:1467] Processing CreateTablet for tablet f9c42e0ae4dc4e4f96cbb9161babbbce (DEFAULT_TABLE table=test-workload [id=d2f0f25ba75b4b18b99f7ede77394cd1]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 13:59:40.393023 1325 tablet_service.cc:1467] Processing CreateTablet for tablet 55b1c1f11b1449d8954ef0c95a95b6da (DEFAULT_TABLE table=test-workload [id=d2f0f25ba75b4b18b99f7ede77394cd1]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 13:59:40.393393 1408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a57d4e298464ad580027819241d4735. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.393601 1327 tablet_service.cc:1467] Processing CreateTablet for tablet 9f1be05bf52a4de19cc2691aa7f73aa6 (DEFAULT_TABLE table=test-workload [id=d2f0f25ba75b4b18b99f7ede77394cd1]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 13:59:40.393678 1324 tablet_service.cc:1467] Processing CreateTablet for tablet 2cad5291c5eb4f12a6bbcbcfdb37a0ed (DEFAULT_TABLE table=test-workload [id=d2f0f25ba75b4b18b99f7ede77394cd1]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 13:59:40.393913 1328 tablet_service.cc:1467] Processing CreateTablet for tablet 3b5648798faa4b2997198f954af3d4e8 (DEFAULT_TABLE table=test-workload [id=d2f0f25ba75b4b18b99f7ede77394cd1]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 13:59:40.393925 1323 tablet_service.cc:1467] Processing CreateTablet for tablet fb8f611a150140b4827890f46869b33f (DEFAULT_TABLE table=test-workload [id=d2f0f25ba75b4b18b99f7ede77394cd1]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 13:59:40.395565 1464 tablet_bootstrap.cc:492] T 2a57d4e298464ad580027819241d4735 P 0c0b5fc6c9894f7ca9a083e986452dff: Bootstrap starting.
I20240502 13:59:40.395712 1407 tablet_service.cc:1467] Processing CreateTablet for tablet 7a8684e4da8f46b6acace6ccac46991f (DEFAULT_TABLE table=test-workload [id=d2f0f25ba75b4b18b99f7ede77394cd1]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 13:59:40.395932 1407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a8684e4da8f46b6acace6ccac46991f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.396049 1401 tablet_service.cc:1467] Processing CreateTablet for tablet fb8f611a150140b4827890f46869b33f (DEFAULT_TABLE table=test-workload [id=d2f0f25ba75b4b18b99f7ede77394cd1]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 13:59:40.396261 1254 tablet_service.cc:1467] Processing CreateTablet for tablet 2a57d4e298464ad580027819241d4735 (DEFAULT_TABLE table=test-workload [id=d2f0f25ba75b4b18b99f7ede77394cd1]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 13:59:40.396499 1254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a57d4e298464ad580027819241d4735. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.396481 1402 tablet_service.cc:1467] Processing CreateTablet for tablet 2cad5291c5eb4f12a6bbcbcfdb37a0ed (DEFAULT_TABLE table=test-workload [id=d2f0f25ba75b4b18b99f7ede77394cd1]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 13:59:40.396879 1403 tablet_service.cc:1467] Processing CreateTablet for tablet 55b1c1f11b1449d8954ef0c95a95b6da (DEFAULT_TABLE table=test-workload [id=d2f0f25ba75b4b18b99f7ede77394cd1]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 13:59:40.397053 1464 tablet_bootstrap.cc:654] T 2a57d4e298464ad580027819241d4735 P 0c0b5fc6c9894f7ca9a083e986452dff: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.397277 1404 tablet_service.cc:1467] Processing CreateTablet for tablet f9c42e0ae4dc4e4f96cbb9161babbbce (DEFAULT_TABLE table=test-workload [id=d2f0f25ba75b4b18b99f7ede77394cd1]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 13:59:40.397655 1405 tablet_service.cc:1467] Processing CreateTablet for tablet 9f1be05bf52a4de19cc2691aa7f73aa6 (DEFAULT_TABLE table=test-workload [id=d2f0f25ba75b4b18b99f7ede77394cd1]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 13:59:40.398054 1406 tablet_service.cc:1467] Processing CreateTablet for tablet 3b5648798faa4b2997198f954af3d4e8 (DEFAULT_TABLE table=test-workload [id=d2f0f25ba75b4b18b99f7ede77394cd1]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 13:59:40.393160 1329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a8684e4da8f46b6acace6ccac46991f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.398530 1466 tablet_bootstrap.cc:492] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc: Bootstrap starting.
I20240502 13:59:40.398658 1253 tablet_service.cc:1467] Processing CreateTablet for tablet 7a8684e4da8f46b6acace6ccac46991f (DEFAULT_TABLE table=test-workload [id=d2f0f25ba75b4b18b99f7ede77394cd1]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 13:59:40.398870 1253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a8684e4da8f46b6acace6ccac46991f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.399358 1466 tablet_bootstrap.cc:654] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.400939 1466 tablet_bootstrap.cc:492] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc: No bootstrap required, opened a new log
I20240502 13:59:40.400979 1250 tablet_service.cc:1467] Processing CreateTablet for tablet f9c42e0ae4dc4e4f96cbb9161babbbce (DEFAULT_TABLE table=test-workload [id=d2f0f25ba75b4b18b99f7ede77394cd1]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 13:59:40.401051 1466 ts_tablet_manager.cc:1367] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 13:59:40.401221 1250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9c42e0ae4dc4e4f96cbb9161babbbce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.401455 1466 raft_consensus.cc:348] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } } peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } }
I20240502 13:59:40.401556 1466 raft_consensus.cc:374] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.401590 1466 raft_consensus.cc:729] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb3125d57f2741698f075887542fc1fc, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.401731 1466 consensus_queue.cc:260] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } } peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } }
I20240502 13:59:40.402052 1466 ts_tablet_manager.cc:1398] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:40.402230 1466 tablet_bootstrap.cc:492] T 7a8684e4da8f46b6acace6ccac46991f P fb3125d57f2741698f075887542fc1fc: Bootstrap starting.
I20240502 13:59:40.403069 1466 tablet_bootstrap.cc:654] T 7a8684e4da8f46b6acace6ccac46991f P fb3125d57f2741698f075887542fc1fc: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.403169 1249 tablet_service.cc:1467] Processing CreateTablet for tablet 55b1c1f11b1449d8954ef0c95a95b6da (DEFAULT_TABLE table=test-workload [id=d2f0f25ba75b4b18b99f7ede77394cd1]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 13:59:40.403393 1249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55b1c1f11b1449d8954ef0c95a95b6da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.404218 1466 tablet_bootstrap.cc:492] T 7a8684e4da8f46b6acace6ccac46991f P fb3125d57f2741698f075887542fc1fc: No bootstrap required, opened a new log
I20240502 13:59:40.404299 1466 ts_tablet_manager.cc:1367] T 7a8684e4da8f46b6acace6ccac46991f P fb3125d57f2741698f075887542fc1fc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:40.404687 1466 raft_consensus.cc:348] T 7a8684e4da8f46b6acace6ccac46991f P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.404808 1466 raft_consensus.cc:374] T 7a8684e4da8f46b6acace6ccac46991f P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.404842 1466 raft_consensus.cc:729] T 7a8684e4da8f46b6acace6ccac46991f P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb3125d57f2741698f075887542fc1fc, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.404942 1466 consensus_queue.cc:260] T 7a8684e4da8f46b6acace6ccac46991f P fb3125d57f2741698f075887542fc1fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.405253 1466 ts_tablet_manager.cc:1398] T 7a8684e4da8f46b6acace6ccac46991f P fb3125d57f2741698f075887542fc1fc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:40.405401 1466 tablet_bootstrap.cc:492] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc: Bootstrap starting.
I20240502 13:59:40.405350 1248 tablet_service.cc:1467] Processing CreateTablet for tablet 2cad5291c5eb4f12a6bbcbcfdb37a0ed (DEFAULT_TABLE table=test-workload [id=d2f0f25ba75b4b18b99f7ede77394cd1]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 13:59:40.405575 1248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cad5291c5eb4f12a6bbcbcfdb37a0ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.406174 1466 tablet_bootstrap.cc:654] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.407327 1466 tablet_bootstrap.cc:492] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc: No bootstrap required, opened a new log
I20240502 13:59:40.407404 1466 ts_tablet_manager.cc:1367] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:40.407493 1247 tablet_service.cc:1467] Processing CreateTablet for tablet fb8f611a150140b4827890f46869b33f (DEFAULT_TABLE table=test-workload [id=d2f0f25ba75b4b18b99f7ede77394cd1]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 13:59:40.407716 1247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb8f611a150140b4827890f46869b33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.407763 1466 raft_consensus.cc:348] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.407855 1466 raft_consensus.cc:374] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.407888 1466 raft_consensus.cc:729] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb3125d57f2741698f075887542fc1fc, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.407992 1466 consensus_queue.cc:260] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.408243 1466 ts_tablet_manager.cc:1398] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:40.408380 1466 tablet_bootstrap.cc:492] T 55b1c1f11b1449d8954ef0c95a95b6da P fb3125d57f2741698f075887542fc1fc: Bootstrap starting.
I20240502 13:59:40.408993 1251 tablet_service.cc:1467] Processing CreateTablet for tablet 9f1be05bf52a4de19cc2691aa7f73aa6 (DEFAULT_TABLE table=test-workload [id=d2f0f25ba75b4b18b99f7ede77394cd1]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 13:59:40.409186 1466 tablet_bootstrap.cc:654] T 55b1c1f11b1449d8954ef0c95a95b6da P fb3125d57f2741698f075887542fc1fc: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.409216 1251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f1be05bf52a4de19cc2691aa7f73aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.400688 1252 tablet_service.cc:1467] Processing CreateTablet for tablet 3b5648798faa4b2997198f954af3d4e8 (DEFAULT_TABLE table=test-workload [id=d2f0f25ba75b4b18b99f7ede77394cd1]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 13:59:40.411057 1252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b5648798faa4b2997198f954af3d4e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.396276 1401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb8f611a150140b4827890f46869b33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.412901 1404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9c42e0ae4dc4e4f96cbb9161babbbce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.413014 1324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cad5291c5eb4f12a6bbcbcfdb37a0ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.413035 1406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b5648798faa4b2997198f954af3d4e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.413182 1402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cad5291c5eb4f12a6bbcbcfdb37a0ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.413302 1403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55b1c1f11b1449d8954ef0c95a95b6da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.413780 1325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55b1c1f11b1449d8954ef0c95a95b6da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.415345 1405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f1be05bf52a4de19cc2691aa7f73aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.415958 1330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a57d4e298464ad580027819241d4735. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.416843 1326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9c42e0ae4dc4e4f96cbb9161babbbce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.418521 1327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f1be05bf52a4de19cc2691aa7f73aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.419776 1323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb8f611a150140b4827890f46869b33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.420876 1328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b5648798faa4b2997198f954af3d4e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.425113 1468 tablet_bootstrap.cc:492] T 7a8684e4da8f46b6acace6ccac46991f P 8feeac9977814b8b901607ec4071efdf: Bootstrap starting.
I20240502 13:59:40.425916 1466 tablet_bootstrap.cc:492] T 55b1c1f11b1449d8954ef0c95a95b6da P fb3125d57f2741698f075887542fc1fc: No bootstrap required, opened a new log
I20240502 13:59:40.425961 1468 tablet_bootstrap.cc:654] T 7a8684e4da8f46b6acace6ccac46991f P 8feeac9977814b8b901607ec4071efdf: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.425989 1464 tablet_bootstrap.cc:492] T 2a57d4e298464ad580027819241d4735 P 0c0b5fc6c9894f7ca9a083e986452dff: No bootstrap required, opened a new log
I20240502 13:59:40.425994 1466 ts_tablet_manager.cc:1367] T 55b1c1f11b1449d8954ef0c95a95b6da P fb3125d57f2741698f075887542fc1fc: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.000s
I20240502 13:59:40.426081 1464 ts_tablet_manager.cc:1367] T 2a57d4e298464ad580027819241d4735 P 0c0b5fc6c9894f7ca9a083e986452dff: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.000s
I20240502 13:59:40.426350 1466 raft_consensus.cc:348] T 55b1c1f11b1449d8954ef0c95a95b6da P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.426446 1466 raft_consensus.cc:374] T 55b1c1f11b1449d8954ef0c95a95b6da P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.426477 1466 raft_consensus.cc:729] T 55b1c1f11b1449d8954ef0c95a95b6da P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb3125d57f2741698f075887542fc1fc, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.426478 1464 raft_consensus.cc:348] T 2a57d4e298464ad580027819241d4735 P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } } peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } }
I20240502 13:59:40.426596 1466 consensus_queue.cc:260] T 55b1c1f11b1449d8954ef0c95a95b6da P fb3125d57f2741698f075887542fc1fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.426604 1464 raft_consensus.cc:374] T 2a57d4e298464ad580027819241d4735 P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.426728 1464 raft_consensus.cc:729] T 2a57d4e298464ad580027819241d4735 P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c0b5fc6c9894f7ca9a083e986452dff, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.426848 1464 consensus_queue.cc:260] T 2a57d4e298464ad580027819241d4735 P 0c0b5fc6c9894f7ca9a083e986452dff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } } peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } }
I20240502 13:59:40.426894 1466 ts_tablet_manager.cc:1398] T 55b1c1f11b1449d8954ef0c95a95b6da P fb3125d57f2741698f075887542fc1fc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:40.427024 1468 tablet_bootstrap.cc:492] T 7a8684e4da8f46b6acace6ccac46991f P 8feeac9977814b8b901607ec4071efdf: No bootstrap required, opened a new log
I20240502 13:59:40.427065 1466 tablet_bootstrap.cc:492] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc: Bootstrap starting.
I20240502 13:59:40.427090 1468 ts_tablet_manager.cc:1367] T 7a8684e4da8f46b6acace6ccac46991f P 8feeac9977814b8b901607ec4071efdf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:40.427124 1464 ts_tablet_manager.cc:1398] T 2a57d4e298464ad580027819241d4735 P 0c0b5fc6c9894f7ca9a083e986452dff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:40.427277 1464 tablet_bootstrap.cc:492] T 7a8684e4da8f46b6acace6ccac46991f P 0c0b5fc6c9894f7ca9a083e986452dff: Bootstrap starting.
I20240502 13:59:40.427470 1468 raft_consensus.cc:348] T 7a8684e4da8f46b6acace6ccac46991f P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.427577 1468 raft_consensus.cc:374] T 7a8684e4da8f46b6acace6ccac46991f P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.427618 1468 raft_consensus.cc:729] T 7a8684e4da8f46b6acace6ccac46991f P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8feeac9977814b8b901607ec4071efdf, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.427727 1468 consensus_queue.cc:260] T 7a8684e4da8f46b6acace6ccac46991f P 8feeac9977814b8b901607ec4071efdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.427829 1466 tablet_bootstrap.cc:654] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.427994 1464 tablet_bootstrap.cc:654] T 7a8684e4da8f46b6acace6ccac46991f P 0c0b5fc6c9894f7ca9a083e986452dff: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.428046 1468 ts_tablet_manager.cc:1398] T 7a8684e4da8f46b6acace6ccac46991f P 8feeac9977814b8b901607ec4071efdf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:40.428191 1468 tablet_bootstrap.cc:492] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf: Bootstrap starting.
I20240502 13:59:40.428965 1468 tablet_bootstrap.cc:654] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.430227 1468 tablet_bootstrap.cc:492] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf: No bootstrap required, opened a new log
I20240502 13:59:40.430317 1468 ts_tablet_manager.cc:1367] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:40.430327 1464 tablet_bootstrap.cc:492] T 7a8684e4da8f46b6acace6ccac46991f P 0c0b5fc6c9894f7ca9a083e986452dff: No bootstrap required, opened a new log
I20240502 13:59:40.430402 1464 ts_tablet_manager.cc:1367] T 7a8684e4da8f46b6acace6ccac46991f P 0c0b5fc6c9894f7ca9a083e986452dff: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 13:59:40.430574 1466 tablet_bootstrap.cc:492] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc: No bootstrap required, opened a new log
I20240502 13:59:40.430692 1466 ts_tablet_manager.cc:1367] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 13:59:40.430730 1468 raft_consensus.cc:348] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.430824 1468 raft_consensus.cc:374] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.430868 1468 raft_consensus.cc:729] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8feeac9977814b8b901607ec4071efdf, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.430981 1468 consensus_queue.cc:260] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.431066 1466 raft_consensus.cc:348] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.431143 1466 raft_consensus.cc:374] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.431180 1466 raft_consensus.cc:729] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb3125d57f2741698f075887542fc1fc, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.431217 1468 ts_tablet_manager.cc:1398] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:40.431310 1466 consensus_queue.cc:260] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.430774 1464 raft_consensus.cc:348] T 7a8684e4da8f46b6acace6ccac46991f P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.431412 1464 raft_consensus.cc:374] T 7a8684e4da8f46b6acace6ccac46991f P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.431442 1464 raft_consensus.cc:729] T 7a8684e4da8f46b6acace6ccac46991f P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c0b5fc6c9894f7ca9a083e986452dff, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.431540 1464 consensus_queue.cc:260] T 7a8684e4da8f46b6acace6ccac46991f P 0c0b5fc6c9894f7ca9a083e986452dff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.431610 1466 ts_tablet_manager.cc:1398] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:40.431746 1464 ts_tablet_manager.cc:1398] T 7a8684e4da8f46b6acace6ccac46991f P 0c0b5fc6c9894f7ca9a083e986452dff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:40.431888 1464 tablet_bootstrap.cc:492] T f9c42e0ae4dc4e4f96cbb9161babbbce P 0c0b5fc6c9894f7ca9a083e986452dff: Bootstrap starting.
I20240502 13:59:40.432562 1464 tablet_bootstrap.cc:654] T f9c42e0ae4dc4e4f96cbb9161babbbce P 0c0b5fc6c9894f7ca9a083e986452dff: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.431367 1468 tablet_bootstrap.cc:492] T f9c42e0ae4dc4e4f96cbb9161babbbce P 8feeac9977814b8b901607ec4071efdf: Bootstrap starting.
I20240502 13:59:40.433847 1468 tablet_bootstrap.cc:654] T f9c42e0ae4dc4e4f96cbb9161babbbce P 8feeac9977814b8b901607ec4071efdf: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.434481 1464 tablet_bootstrap.cc:492] T f9c42e0ae4dc4e4f96cbb9161babbbce P 0c0b5fc6c9894f7ca9a083e986452dff: No bootstrap required, opened a new log
I20240502 13:59:40.434569 1464 ts_tablet_manager.cc:1367] T f9c42e0ae4dc4e4f96cbb9161babbbce P 0c0b5fc6c9894f7ca9a083e986452dff: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 13:59:40.434821 1468 tablet_bootstrap.cc:492] T f9c42e0ae4dc4e4f96cbb9161babbbce P 8feeac9977814b8b901607ec4071efdf: No bootstrap required, opened a new log
I20240502 13:59:40.434896 1468 ts_tablet_manager.cc:1367] T f9c42e0ae4dc4e4f96cbb9161babbbce P 8feeac9977814b8b901607ec4071efdf: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 13:59:40.434921 1464 raft_consensus.cc:348] T f9c42e0ae4dc4e4f96cbb9161babbbce P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.435014 1464 raft_consensus.cc:374] T f9c42e0ae4dc4e4f96cbb9161babbbce P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.435048 1464 raft_consensus.cc:729] T f9c42e0ae4dc4e4f96cbb9161babbbce P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c0b5fc6c9894f7ca9a083e986452dff, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.435158 1464 consensus_queue.cc:260] T f9c42e0ae4dc4e4f96cbb9161babbbce P 0c0b5fc6c9894f7ca9a083e986452dff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.435386 1468 raft_consensus.cc:348] T f9c42e0ae4dc4e4f96cbb9161babbbce P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.431751 1466 tablet_bootstrap.cc:492] T 9f1be05bf52a4de19cc2691aa7f73aa6 P fb3125d57f2741698f075887542fc1fc: Bootstrap starting.
I20240502 13:59:40.435487 1468 raft_consensus.cc:374] T f9c42e0ae4dc4e4f96cbb9161babbbce P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.435529 1468 raft_consensus.cc:729] T f9c42e0ae4dc4e4f96cbb9161babbbce P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8feeac9977814b8b901607ec4071efdf, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.435655 1468 consensus_queue.cc:260] T f9c42e0ae4dc4e4f96cbb9161babbbce P 8feeac9977814b8b901607ec4071efdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.435976 1468 ts_tablet_manager.cc:1398] T f9c42e0ae4dc4e4f96cbb9161babbbce P 8feeac9977814b8b901607ec4071efdf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:40.436213 1466 tablet_bootstrap.cc:654] T 9f1be05bf52a4de19cc2691aa7f73aa6 P fb3125d57f2741698f075887542fc1fc: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.436569 1468 tablet_bootstrap.cc:492] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 8feeac9977814b8b901607ec4071efdf: Bootstrap starting.
I20240502 13:59:40.435431 1464 ts_tablet_manager.cc:1398] T f9c42e0ae4dc4e4f96cbb9161babbbce P 0c0b5fc6c9894f7ca9a083e986452dff: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:40.436811 1464 tablet_bootstrap.cc:492] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 0c0b5fc6c9894f7ca9a083e986452dff: Bootstrap starting.
I20240502 13:59:40.437577 1464 tablet_bootstrap.cc:654] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 0c0b5fc6c9894f7ca9a083e986452dff: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.438256 1466 tablet_bootstrap.cc:492] T 9f1be05bf52a4de19cc2691aa7f73aa6 P fb3125d57f2741698f075887542fc1fc: No bootstrap required, opened a new log
I20240502 13:59:40.438382 1466 ts_tablet_manager.cc:1367] T 9f1be05bf52a4de19cc2691aa7f73aa6 P fb3125d57f2741698f075887542fc1fc: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 13:59:40.438573 1464 tablet_bootstrap.cc:492] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 0c0b5fc6c9894f7ca9a083e986452dff: No bootstrap required, opened a new log
I20240502 13:59:40.438653 1464 ts_tablet_manager.cc:1367] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 0c0b5fc6c9894f7ca9a083e986452dff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:40.438689 1468 tablet_bootstrap.cc:654] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 8feeac9977814b8b901607ec4071efdf: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.438925 1466 raft_consensus.cc:348] T 9f1be05bf52a4de19cc2691aa7f73aa6 P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.439002 1464 raft_consensus.cc:348] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.439081 1466 raft_consensus.cc:374] T 9f1be05bf52a4de19cc2691aa7f73aa6 P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.439098 1464 raft_consensus.cc:374] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.439119 1466 raft_consensus.cc:729] T 9f1be05bf52a4de19cc2691aa7f73aa6 P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb3125d57f2741698f075887542fc1fc, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.439129 1464 raft_consensus.cc:729] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c0b5fc6c9894f7ca9a083e986452dff, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.439252 1466 consensus_queue.cc:260] T 9f1be05bf52a4de19cc2691aa7f73aa6 P fb3125d57f2741698f075887542fc1fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.439252 1464 consensus_queue.cc:260] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 0c0b5fc6c9894f7ca9a083e986452dff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.439535 1468 tablet_bootstrap.cc:492] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 8feeac9977814b8b901607ec4071efdf: No bootstrap required, opened a new log
I20240502 13:59:40.439582 1464 ts_tablet_manager.cc:1398] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 0c0b5fc6c9894f7ca9a083e986452dff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:40.439610 1468 ts_tablet_manager.cc:1367] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 8feeac9977814b8b901607ec4071efdf: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 13:59:40.439729 1464 tablet_bootstrap.cc:492] T 3b5648798faa4b2997198f954af3d4e8 P 0c0b5fc6c9894f7ca9a083e986452dff: Bootstrap starting.
I20240502 13:59:40.439838 1466 ts_tablet_manager.cc:1398] T 9f1be05bf52a4de19cc2691aa7f73aa6 P fb3125d57f2741698f075887542fc1fc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:40.439983 1466 tablet_bootstrap.cc:492] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc: Bootstrap starting.
I20240502 13:59:40.440069 1468 raft_consensus.cc:348] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.440155 1468 raft_consensus.cc:374] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.440184 1468 raft_consensus.cc:729] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8feeac9977814b8b901607ec4071efdf, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.440311 1468 consensus_queue.cc:260] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 8feeac9977814b8b901607ec4071efdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.440537 1468 ts_tablet_manager.cc:1398] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 8feeac9977814b8b901607ec4071efdf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:40.440793 1466 tablet_bootstrap.cc:654] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.441033 1468 tablet_bootstrap.cc:492] T fb8f611a150140b4827890f46869b33f P 8feeac9977814b8b901607ec4071efdf: Bootstrap starting.
I20240502 13:59:40.442407 1468 tablet_bootstrap.cc:654] T fb8f611a150140b4827890f46869b33f P 8feeac9977814b8b901607ec4071efdf: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.442605 1464 tablet_bootstrap.cc:654] T 3b5648798faa4b2997198f954af3d4e8 P 0c0b5fc6c9894f7ca9a083e986452dff: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.443233 1468 tablet_bootstrap.cc:492] T fb8f611a150140b4827890f46869b33f P 8feeac9977814b8b901607ec4071efdf: No bootstrap required, opened a new log
I20240502 13:59:40.443305 1468 ts_tablet_manager.cc:1367] T fb8f611a150140b4827890f46869b33f P 8feeac9977814b8b901607ec4071efdf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:40.443461 1464 tablet_bootstrap.cc:492] T 3b5648798faa4b2997198f954af3d4e8 P 0c0b5fc6c9894f7ca9a083e986452dff: No bootstrap required, opened a new log
I20240502 13:59:40.443531 1464 ts_tablet_manager.cc:1367] T 3b5648798faa4b2997198f954af3d4e8 P 0c0b5fc6c9894f7ca9a083e986452dff: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 13:59:40.443645 1468 raft_consensus.cc:348] T fb8f611a150140b4827890f46869b33f P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.443720 1468 raft_consensus.cc:374] T fb8f611a150140b4827890f46869b33f P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.443749 1468 raft_consensus.cc:729] T fb8f611a150140b4827890f46869b33f P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8feeac9977814b8b901607ec4071efdf, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.443854 1468 consensus_queue.cc:260] T fb8f611a150140b4827890f46869b33f P 8feeac9977814b8b901607ec4071efdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.443852 1464 raft_consensus.cc:348] T 3b5648798faa4b2997198f954af3d4e8 P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.443982 1464 raft_consensus.cc:374] T 3b5648798faa4b2997198f954af3d4e8 P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.444020 1464 raft_consensus.cc:729] T 3b5648798faa4b2997198f954af3d4e8 P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c0b5fc6c9894f7ca9a083e986452dff, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.444099 1468 ts_tablet_manager.cc:1398] T fb8f611a150140b4827890f46869b33f P 8feeac9977814b8b901607ec4071efdf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:40.444124 1464 consensus_queue.cc:260] T 3b5648798faa4b2997198f954af3d4e8 P 0c0b5fc6c9894f7ca9a083e986452dff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.444243 1468 tablet_bootstrap.cc:492] T 3b5648798faa4b2997198f954af3d4e8 P 8feeac9977814b8b901607ec4071efdf: Bootstrap starting.
I20240502 13:59:40.444345 1464 ts_tablet_manager.cc:1398] T 3b5648798faa4b2997198f954af3d4e8 P 0c0b5fc6c9894f7ca9a083e986452dff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:40.444559 1464 tablet_bootstrap.cc:492] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 0c0b5fc6c9894f7ca9a083e986452dff: Bootstrap starting.
I20240502 13:59:40.444991 1468 tablet_bootstrap.cc:654] T 3b5648798faa4b2997198f954af3d4e8 P 8feeac9977814b8b901607ec4071efdf: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.445415 1464 tablet_bootstrap.cc:654] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 0c0b5fc6c9894f7ca9a083e986452dff: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.445957 1466 tablet_bootstrap.cc:492] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc: No bootstrap required, opened a new log
I20240502 13:59:40.445966 1468 tablet_bootstrap.cc:492] T 3b5648798faa4b2997198f954af3d4e8 P 8feeac9977814b8b901607ec4071efdf: No bootstrap required, opened a new log
I20240502 13:59:40.446044 1466 ts_tablet_manager.cc:1367] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240502 13:59:40.446069 1468 ts_tablet_manager.cc:1367] T 3b5648798faa4b2997198f954af3d4e8 P 8feeac9977814b8b901607ec4071efdf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:40.446468 1464 tablet_bootstrap.cc:492] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 0c0b5fc6c9894f7ca9a083e986452dff: No bootstrap required, opened a new log
I20240502 13:59:40.446426 1466 raft_consensus.cc:348] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.446426 1468 raft_consensus.cc:348] T 3b5648798faa4b2997198f954af3d4e8 P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.446557 1466 raft_consensus.cc:374] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.446588 1466 raft_consensus.cc:729] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb3125d57f2741698f075887542fc1fc, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.446581 1468 raft_consensus.cc:374] T 3b5648798faa4b2997198f954af3d4e8 P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.446581 1464 ts_tablet_manager.cc:1367] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 0c0b5fc6c9894f7ca9a083e986452dff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:40.446637 1468 raft_consensus.cc:729] T 3b5648798faa4b2997198f954af3d4e8 P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8feeac9977814b8b901607ec4071efdf, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.446687 1466 consensus_queue.cc:260] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.446723 1468 consensus_queue.cc:260] T 3b5648798faa4b2997198f954af3d4e8 P 8feeac9977814b8b901607ec4071efdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.446956 1468 ts_tablet_manager.cc:1398] T 3b5648798faa4b2997198f954af3d4e8 P 8feeac9977814b8b901607ec4071efdf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:40.447026 1464 raft_consensus.cc:348] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.447095 1468 tablet_bootstrap.cc:492] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 8feeac9977814b8b901607ec4071efdf: Bootstrap starting.
I20240502 13:59:40.447115 1464 raft_consensus.cc:374] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.447149 1464 raft_consensus.cc:729] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c0b5fc6c9894f7ca9a083e986452dff, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.447263 1464 consensus_queue.cc:260] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 0c0b5fc6c9894f7ca9a083e986452dff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.447348 1466 ts_tablet_manager.cc:1398] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:40.447502 1466 tablet_bootstrap.cc:492] T fb8f611a150140b4827890f46869b33f P fb3125d57f2741698f075887542fc1fc: Bootstrap starting.
I20240502 13:59:40.447546 1464 ts_tablet_manager.cc:1398] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 0c0b5fc6c9894f7ca9a083e986452dff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:40.447691 1464 tablet_bootstrap.cc:492] T 55b1c1f11b1449d8954ef0c95a95b6da P 0c0b5fc6c9894f7ca9a083e986452dff: Bootstrap starting.
I20240502 13:59:40.447875 1468 tablet_bootstrap.cc:654] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 8feeac9977814b8b901607ec4071efdf: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.448263 1466 tablet_bootstrap.cc:654] T fb8f611a150140b4827890f46869b33f P fb3125d57f2741698f075887542fc1fc: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.448503 1464 tablet_bootstrap.cc:654] T 55b1c1f11b1449d8954ef0c95a95b6da P 0c0b5fc6c9894f7ca9a083e986452dff: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.449486 1468 tablet_bootstrap.cc:492] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 8feeac9977814b8b901607ec4071efdf: No bootstrap required, opened a new log
I20240502 13:59:40.449569 1468 ts_tablet_manager.cc:1367] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 8feeac9977814b8b901607ec4071efdf: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 13:59:40.449658 1464 tablet_bootstrap.cc:492] T 55b1c1f11b1449d8954ef0c95a95b6da P 0c0b5fc6c9894f7ca9a083e986452dff: No bootstrap required, opened a new log
I20240502 13:59:40.449726 1464 ts_tablet_manager.cc:1367] T 55b1c1f11b1449d8954ef0c95a95b6da P 0c0b5fc6c9894f7ca9a083e986452dff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 13:59:40.449947 1468 raft_consensus.cc:348] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.450032 1468 raft_consensus.cc:374] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.450069 1468 raft_consensus.cc:729] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8feeac9977814b8b901607ec4071efdf, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.450093 1464 raft_consensus.cc:348] T 55b1c1f11b1449d8954ef0c95a95b6da P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.450182 1464 raft_consensus.cc:374] T 55b1c1f11b1449d8954ef0c95a95b6da P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.450213 1464 raft_consensus.cc:729] T 55b1c1f11b1449d8954ef0c95a95b6da P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c0b5fc6c9894f7ca9a083e986452dff, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.450208 1468 consensus_queue.cc:260] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 8feeac9977814b8b901607ec4071efdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.450376 1464 consensus_queue.cc:260] T 55b1c1f11b1449d8954ef0c95a95b6da P 0c0b5fc6c9894f7ca9a083e986452dff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.450469 1468 ts_tablet_manager.cc:1398] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 8feeac9977814b8b901607ec4071efdf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:40.450646 1468 tablet_bootstrap.cc:492] T 2a57d4e298464ad580027819241d4735 P 8feeac9977814b8b901607ec4071efdf: Bootstrap starting.
I20240502 13:59:40.450888 1464 ts_tablet_manager.cc:1398] T 55b1c1f11b1449d8954ef0c95a95b6da P 0c0b5fc6c9894f7ca9a083e986452dff: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:40.451053 1464 tablet_bootstrap.cc:492] T fb8f611a150140b4827890f46869b33f P 0c0b5fc6c9894f7ca9a083e986452dff: Bootstrap starting.
I20240502 13:59:40.451478 1468 tablet_bootstrap.cc:654] T 2a57d4e298464ad580027819241d4735 P 8feeac9977814b8b901607ec4071efdf: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.452275 1464 tablet_bootstrap.cc:654] T fb8f611a150140b4827890f46869b33f P 0c0b5fc6c9894f7ca9a083e986452dff: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.452457 1468 tablet_bootstrap.cc:492] T 2a57d4e298464ad580027819241d4735 P 8feeac9977814b8b901607ec4071efdf: No bootstrap required, opened a new log
I20240502 13:59:40.452543 1468 ts_tablet_manager.cc:1367] T 2a57d4e298464ad580027819241d4735 P 8feeac9977814b8b901607ec4071efdf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:40.452981 1468 raft_consensus.cc:348] T 2a57d4e298464ad580027819241d4735 P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } } peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } }
I20240502 13:59:40.453083 1468 raft_consensus.cc:374] T 2a57d4e298464ad580027819241d4735 P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.453121 1468 raft_consensus.cc:729] T 2a57d4e298464ad580027819241d4735 P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8feeac9977814b8b901607ec4071efdf, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.453245 1468 consensus_queue.cc:260] T 2a57d4e298464ad580027819241d4735 P 8feeac9977814b8b901607ec4071efdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } } peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } }
I20240502 13:59:40.453282 1464 tablet_bootstrap.cc:492] T fb8f611a150140b4827890f46869b33f P 0c0b5fc6c9894f7ca9a083e986452dff: No bootstrap required, opened a new log
I20240502 13:59:40.453282 1466 tablet_bootstrap.cc:492] T fb8f611a150140b4827890f46869b33f P fb3125d57f2741698f075887542fc1fc: No bootstrap required, opened a new log
I20240502 13:59:40.453382 1466 ts_tablet_manager.cc:1367] T fb8f611a150140b4827890f46869b33f P fb3125d57f2741698f075887542fc1fc: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 13:59:40.453490 1468 ts_tablet_manager.cc:1398] T 2a57d4e298464ad580027819241d4735 P 8feeac9977814b8b901607ec4071efdf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:40.453519 1464 ts_tablet_manager.cc:1367] T fb8f611a150140b4827890f46869b33f P 0c0b5fc6c9894f7ca9a083e986452dff: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 13:59:40.453740 1466 raft_consensus.cc:348] T fb8f611a150140b4827890f46869b33f P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.453836 1466 raft_consensus.cc:374] T fb8f611a150140b4827890f46869b33f P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.453869 1466 raft_consensus.cc:729] T fb8f611a150140b4827890f46869b33f P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb3125d57f2741698f075887542fc1fc, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.453931 1464 raft_consensus.cc:348] T fb8f611a150140b4827890f46869b33f P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.454007 1464 raft_consensus.cc:374] T fb8f611a150140b4827890f46869b33f P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.454036 1464 raft_consensus.cc:729] T fb8f611a150140b4827890f46869b33f P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c0b5fc6c9894f7ca9a083e986452dff, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.454018 1466 consensus_queue.cc:260] T fb8f611a150140b4827890f46869b33f P fb3125d57f2741698f075887542fc1fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.454123 1464 consensus_queue.cc:260] T fb8f611a150140b4827890f46869b33f P 0c0b5fc6c9894f7ca9a083e986452dff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.454624 1464 ts_tablet_manager.cc:1398] T fb8f611a150140b4827890f46869b33f P 0c0b5fc6c9894f7ca9a083e986452dff: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:40.454890 1466 ts_tablet_manager.cc:1398] T fb8f611a150140b4827890f46869b33f P fb3125d57f2741698f075887542fc1fc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:40.501405 1469 raft_consensus.cc:482] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:40.501514 1469 raft_consensus.cc:504] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.502002 1469 leader_election.cc:288] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8feeac9977814b8b901607ec4071efdf (127.0.145.66:45147), 0c0b5fc6c9894f7ca9a083e986452dff (127.0.145.67:41377)
I20240502 13:59:40.505578 1418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cad5291c5eb4f12a6bbcbcfdb37a0ed" candidate_uuid: "fb3125d57f2741698f075887542fc1fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" is_pre_election: true
I20240502 13:59:40.505694 1418 raft_consensus.cc:2454] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb3125d57f2741698f075887542fc1fc in term 0.
I20240502 13:59:40.505728 1340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cad5291c5eb4f12a6bbcbcfdb37a0ed" candidate_uuid: "fb3125d57f2741698f075887542fc1fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8feeac9977814b8b901607ec4071efdf" is_pre_election: true
I20240502 13:59:40.505848 1340 raft_consensus.cc:2454] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb3125d57f2741698f075887542fc1fc in term 0.
I20240502 13:59:40.505964 1227 leader_election.cc:302] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c0b5fc6c9894f7ca9a083e986452dff, fb3125d57f2741698f075887542fc1fc; no voters:
I20240502 13:59:40.506140 1469 raft_consensus.cc:2781] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:40.506189 1469 raft_consensus.cc:482] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:40.506230 1469 raft_consensus.cc:3037] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.507135 1469 raft_consensus.cc:504] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.507396 1469 leader_election.cc:288] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc [CANDIDATE]: Term 1 election: Requested vote from peers 8feeac9977814b8b901607ec4071efdf (127.0.145.66:45147), 0c0b5fc6c9894f7ca9a083e986452dff (127.0.145.67:41377)
I20240502 13:59:40.507648 1340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cad5291c5eb4f12a6bbcbcfdb37a0ed" candidate_uuid: "fb3125d57f2741698f075887542fc1fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8feeac9977814b8b901607ec4071efdf"
I20240502 13:59:40.507705 1418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cad5291c5eb4f12a6bbcbcfdb37a0ed" candidate_uuid: "fb3125d57f2741698f075887542fc1fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c0b5fc6c9894f7ca9a083e986452dff"
I20240502 13:59:40.507735 1340 raft_consensus.cc:3037] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.507810 1418 raft_consensus.cc:3037] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.508471 1340 raft_consensus.cc:2454] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 8feeac9977814b8b901607ec4071efdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb3125d57f2741698f075887542fc1fc in term 1.
I20240502 13:59:40.508562 1418 raft_consensus.cc:2454] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb3125d57f2741698f075887542fc1fc in term 1.
I20240502 13:59:40.508711 1229 leader_election.cc:302] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8feeac9977814b8b901607ec4071efdf, fb3125d57f2741698f075887542fc1fc; no voters:
I20240502 13:59:40.508888 1469 raft_consensus.cc:2781] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:40.508982 1469 raft_consensus.cc:686] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc [term 1 LEADER]: Becoming Leader. State: Replica: fb3125d57f2741698f075887542fc1fc, State: Running, Role: LEADER
I20240502 13:59:40.509109 1469 consensus_queue.cc:237] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.510494 1158 catalog_manager.cc:5484] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc reported cstate change: term changed from 0 to 1, leader changed from <none> to fb3125d57f2741698f075887542fc1fc (127.0.145.65). New cstate: current_term: 1 leader_uuid: "fb3125d57f2741698f075887542fc1fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:40.521273 1477 raft_consensus.cc:482] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:40.521348 1477 raft_consensus.cc:504] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.521665 1477 leader_election.cc:288] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb3125d57f2741698f075887542fc1fc (127.0.145.65:34005), 0c0b5fc6c9894f7ca9a083e986452dff (127.0.145.67:41377)
I20240502 13:59:40.524531 1418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b1c1f11b1449d8954ef0c95a95b6da" candidate_uuid: "8feeac9977814b8b901607ec4071efdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" is_pre_election: true
I20240502 13:59:40.524539 1264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b1c1f11b1449d8954ef0c95a95b6da" candidate_uuid: "8feeac9977814b8b901607ec4071efdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb3125d57f2741698f075887542fc1fc" is_pre_election: true
I20240502 13:59:40.524636 1418 raft_consensus.cc:2454] T 55b1c1f11b1449d8954ef0c95a95b6da P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8feeac9977814b8b901607ec4071efdf in term 0.
I20240502 13:59:40.524686 1264 raft_consensus.cc:2454] T 55b1c1f11b1449d8954ef0c95a95b6da P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8feeac9977814b8b901607ec4071efdf in term 0.
I20240502 13:59:40.524911 1303 leader_election.cc:302] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c0b5fc6c9894f7ca9a083e986452dff, 8feeac9977814b8b901607ec4071efdf; no voters:
I20240502 13:59:40.525054 1477 raft_consensus.cc:2781] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:40.525111 1477 raft_consensus.cc:482] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:40.525147 1477 raft_consensus.cc:3037] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.525751 1477 raft_consensus.cc:504] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.526000 1477 leader_election.cc:288] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf [CANDIDATE]: Term 1 election: Requested vote from peers fb3125d57f2741698f075887542fc1fc (127.0.145.65:34005), 0c0b5fc6c9894f7ca9a083e986452dff (127.0.145.67:41377)
I20240502 13:59:40.526239 1264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b1c1f11b1449d8954ef0c95a95b6da" candidate_uuid: "8feeac9977814b8b901607ec4071efdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb3125d57f2741698f075887542fc1fc"
I20240502 13:59:40.526265 1418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b1c1f11b1449d8954ef0c95a95b6da" candidate_uuid: "8feeac9977814b8b901607ec4071efdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c0b5fc6c9894f7ca9a083e986452dff"
I20240502 13:59:40.526331 1264 raft_consensus.cc:3037] T 55b1c1f11b1449d8954ef0c95a95b6da P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.526351 1418 raft_consensus.cc:3037] T 55b1c1f11b1449d8954ef0c95a95b6da P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.526942 1418 raft_consensus.cc:2454] T 55b1c1f11b1449d8954ef0c95a95b6da P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8feeac9977814b8b901607ec4071efdf in term 1.
I20240502 13:59:40.527053 1264 raft_consensus.cc:2454] T 55b1c1f11b1449d8954ef0c95a95b6da P fb3125d57f2741698f075887542fc1fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8feeac9977814b8b901607ec4071efdf in term 1.
I20240502 13:59:40.527185 1303 leader_election.cc:302] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c0b5fc6c9894f7ca9a083e986452dff, 8feeac9977814b8b901607ec4071efdf; no voters:
I20240502 13:59:40.527359 1477 raft_consensus.cc:2781] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:40.527439 1477 raft_consensus.cc:686] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf [term 1 LEADER]: Becoming Leader. State: Replica: 8feeac9977814b8b901607ec4071efdf, State: Running, Role: LEADER
I20240502 13:59:40.527572 1477 consensus_queue.cc:237] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.528843 1158 catalog_manager.cc:5484] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8feeac9977814b8b901607ec4071efdf (127.0.145.66). New cstate: current_term: 1 leader_uuid: "8feeac9977814b8b901607ec4071efdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:40.532562 1469 raft_consensus.cc:482] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:40.532629 1469 raft_consensus.cc:504] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.532915 1469 leader_election.cc:288] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8feeac9977814b8b901607ec4071efdf (127.0.145.66:45147), 0c0b5fc6c9894f7ca9a083e986452dff (127.0.145.67:41377)
I20240502 13:59:40.533125 1340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c42e0ae4dc4e4f96cbb9161babbbce" candidate_uuid: "fb3125d57f2741698f075887542fc1fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8feeac9977814b8b901607ec4071efdf" is_pre_election: true
I20240502 13:59:40.533147 1418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c42e0ae4dc4e4f96cbb9161babbbce" candidate_uuid: "fb3125d57f2741698f075887542fc1fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" is_pre_election: true
I20240502 13:59:40.533217 1340 raft_consensus.cc:2454] T f9c42e0ae4dc4e4f96cbb9161babbbce P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb3125d57f2741698f075887542fc1fc in term 0.
I20240502 13:59:40.533221 1418 raft_consensus.cc:2454] T f9c42e0ae4dc4e4f96cbb9161babbbce P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb3125d57f2741698f075887542fc1fc in term 0.
I20240502 13:59:40.533421 1229 leader_election.cc:302] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8feeac9977814b8b901607ec4071efdf, fb3125d57f2741698f075887542fc1fc; no voters:
I20240502 13:59:40.533563 1469 raft_consensus.cc:2781] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:40.533618 1469 raft_consensus.cc:482] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:40.533661 1469 raft_consensus.cc:3037] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.534212 1469 raft_consensus.cc:504] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.534487 1469 leader_election.cc:288] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc [CANDIDATE]: Term 1 election: Requested vote from peers 8feeac9977814b8b901607ec4071efdf (127.0.145.66:45147), 0c0b5fc6c9894f7ca9a083e986452dff (127.0.145.67:41377)
I20240502 13:59:40.534722 1340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c42e0ae4dc4e4f96cbb9161babbbce" candidate_uuid: "fb3125d57f2741698f075887542fc1fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8feeac9977814b8b901607ec4071efdf"
I20240502 13:59:40.534725 1418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c42e0ae4dc4e4f96cbb9161babbbce" candidate_uuid: "fb3125d57f2741698f075887542fc1fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c0b5fc6c9894f7ca9a083e986452dff"
I20240502 13:59:40.534816 1340 raft_consensus.cc:3037] T f9c42e0ae4dc4e4f96cbb9161babbbce P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.534818 1418 raft_consensus.cc:3037] T f9c42e0ae4dc4e4f96cbb9161babbbce P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.535439 1418 raft_consensus.cc:2454] T f9c42e0ae4dc4e4f96cbb9161babbbce P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb3125d57f2741698f075887542fc1fc in term 1.
I20240502 13:59:40.535638 1227 leader_election.cc:302] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c0b5fc6c9894f7ca9a083e986452dff, fb3125d57f2741698f075887542fc1fc; no voters:
I20240502 13:59:40.535766 1469 raft_consensus.cc:2781] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:40.535769 1340 raft_consensus.cc:2454] T f9c42e0ae4dc4e4f96cbb9161babbbce P 8feeac9977814b8b901607ec4071efdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb3125d57f2741698f075887542fc1fc in term 1.
I20240502 13:59:40.535836 1469 raft_consensus.cc:686] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc [term 1 LEADER]: Becoming Leader. State: Replica: fb3125d57f2741698f075887542fc1fc, State: Running, Role: LEADER
I20240502 13:59:40.535952 1469 consensus_queue.cc:237] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.537232 1158 catalog_manager.cc:5484] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc reported cstate change: term changed from 0 to 1, leader changed from <none> to fb3125d57f2741698f075887542fc1fc (127.0.145.65). New cstate: current_term: 1 leader_uuid: "fb3125d57f2741698f075887542fc1fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:40.539294 1469 raft_consensus.cc:482] T fb8f611a150140b4827890f46869b33f P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:40.539362 1469 raft_consensus.cc:504] T fb8f611a150140b4827890f46869b33f P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.539649 1469 leader_election.cc:288] T fb8f611a150140b4827890f46869b33f P fb3125d57f2741698f075887542fc1fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8feeac9977814b8b901607ec4071efdf (127.0.145.66:45147), 0c0b5fc6c9894f7ca9a083e986452dff (127.0.145.67:41377)
I20240502 13:59:40.539645 1476 raft_consensus.cc:482] T fb8f611a150140b4827890f46869b33f P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:40.539747 1476 raft_consensus.cc:504] T fb8f611a150140b4827890f46869b33f P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.539884 1340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8f611a150140b4827890f46869b33f" candidate_uuid: "fb3125d57f2741698f075887542fc1fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8feeac9977814b8b901607ec4071efdf" is_pre_election: true
I20240502 13:59:40.539965 1340 raft_consensus.cc:2454] T fb8f611a150140b4827890f46869b33f P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb3125d57f2741698f075887542fc1fc in term 0.
I20240502 13:59:40.540119 1476 leader_election.cc:288] T fb8f611a150140b4827890f46869b33f P 0c0b5fc6c9894f7ca9a083e986452dff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb3125d57f2741698f075887542fc1fc (127.0.145.65:34005), 8feeac9977814b8b901607ec4071efdf (127.0.145.66:45147)
I20240502 13:59:40.540130 1418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8f611a150140b4827890f46869b33f" candidate_uuid: "fb3125d57f2741698f075887542fc1fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" is_pre_election: true
I20240502 13:59:40.540194 1229 leader_election.cc:302] T fb8f611a150140b4827890f46869b33f P fb3125d57f2741698f075887542fc1fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8feeac9977814b8b901607ec4071efdf, fb3125d57f2741698f075887542fc1fc; no voters:
I20240502 13:59:40.540230 1418 raft_consensus.cc:2454] T fb8f611a150140b4827890f46869b33f P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb3125d57f2741698f075887542fc1fc in term 0.
I20240502 13:59:40.540345 1469 raft_consensus.cc:2781] T fb8f611a150140b4827890f46869b33f P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:40.540395 1469 raft_consensus.cc:482] T fb8f611a150140b4827890f46869b33f P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:40.540429 1469 raft_consensus.cc:3037] T fb8f611a150140b4827890f46869b33f P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.541289 1469 raft_consensus.cc:504] T fb8f611a150140b4827890f46869b33f P fb3125d57f2741698f075887542fc1fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.541823 1340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8f611a150140b4827890f46869b33f" candidate_uuid: "fb3125d57f2741698f075887542fc1fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8feeac9977814b8b901607ec4071efdf"
I20240502 13:59:40.541918 1340 raft_consensus.cc:3037] T fb8f611a150140b4827890f46869b33f P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.542392 1418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8f611a150140b4827890f46869b33f" candidate_uuid: "fb3125d57f2741698f075887542fc1fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c0b5fc6c9894f7ca9a083e986452dff"
I20240502 13:59:40.542495 1418 raft_consensus.cc:3037] T fb8f611a150140b4827890f46869b33f P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.542529 1340 raft_consensus.cc:2454] T fb8f611a150140b4827890f46869b33f P 8feeac9977814b8b901607ec4071efdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb3125d57f2741698f075887542fc1fc in term 1.
I20240502 13:59:40.542784 1229 leader_election.cc:302] T fb8f611a150140b4827890f46869b33f P fb3125d57f2741698f075887542fc1fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8feeac9977814b8b901607ec4071efdf, fb3125d57f2741698f075887542fc1fc; no voters:
I20240502 13:59:40.542932 1469 leader_election.cc:288] T fb8f611a150140b4827890f46869b33f P fb3125d57f2741698f075887542fc1fc [CANDIDATE]: Term 1 election: Requested vote from peers 8feeac9977814b8b901607ec4071efdf (127.0.145.66:45147), 0c0b5fc6c9894f7ca9a083e986452dff (127.0.145.67:41377)
I20240502 13:59:40.542953 1471 raft_consensus.cc:2781] T fb8f611a150140b4827890f46869b33f P fb3125d57f2741698f075887542fc1fc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:40.543056 1471 raft_consensus.cc:686] T fb8f611a150140b4827890f46869b33f P fb3125d57f2741698f075887542fc1fc [term 1 LEADER]: Becoming Leader. State: Replica: fb3125d57f2741698f075887542fc1fc, State: Running, Role: LEADER
I20240502 13:59:40.543200 1418 raft_consensus.cc:2454] T fb8f611a150140b4827890f46869b33f P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb3125d57f2741698f075887542fc1fc in term 1.
I20240502 13:59:40.543174 1471 consensus_queue.cc:237] T fb8f611a150140b4827890f46869b33f P fb3125d57f2741698f075887542fc1fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.544230 1264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8f611a150140b4827890f46869b33f" candidate_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb3125d57f2741698f075887542fc1fc" is_pre_election: true
I20240502 13:59:40.544891 1158 catalog_manager.cc:5484] T fb8f611a150140b4827890f46869b33f P fb3125d57f2741698f075887542fc1fc reported cstate change: term changed from 0 to 1, leader changed from <none> to fb3125d57f2741698f075887542fc1fc (127.0.145.65). New cstate: current_term: 1 leader_uuid: "fb3125d57f2741698f075887542fc1fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:40.544976 1340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8f611a150140b4827890f46869b33f" candidate_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8feeac9977814b8b901607ec4071efdf" is_pre_election: true
I20240502 13:59:40.545114 1340 raft_consensus.cc:2379] T fb8f611a150140b4827890f46869b33f P 8feeac9977814b8b901607ec4071efdf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0c0b5fc6c9894f7ca9a083e986452dff in current term 1: Already voted for candidate fb3125d57f2741698f075887542fc1fc in this term.
I20240502 13:59:40.545357 1382 leader_election.cc:302] T fb8f611a150140b4827890f46869b33f P 0c0b5fc6c9894f7ca9a083e986452dff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c0b5fc6c9894f7ca9a083e986452dff; no voters: 8feeac9977814b8b901607ec4071efdf, fb3125d57f2741698f075887542fc1fc
I20240502 13:59:40.545496 1476 raft_consensus.cc:2726] T fb8f611a150140b4827890f46869b33f P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 13:59:40.547931 1291 tablet.cc:2369] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 13:59:40.553658 1471 raft_consensus.cc:482] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:40.553725 1471 raft_consensus.cc:504] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } } peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } }
I20240502 13:59:40.553959 1471 leader_election.cc:288] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8feeac9977814b8b901607ec4071efdf (127.0.145.66:45147), 0c0b5fc6c9894f7ca9a083e986452dff (127.0.145.67:41377)
I20240502 13:59:40.554214 1340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a57d4e298464ad580027819241d4735" candidate_uuid: "fb3125d57f2741698f075887542fc1fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8feeac9977814b8b901607ec4071efdf" is_pre_election: true
I20240502 13:59:40.554219 1418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a57d4e298464ad580027819241d4735" candidate_uuid: "fb3125d57f2741698f075887542fc1fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" is_pre_election: true
I20240502 13:59:40.554313 1340 raft_consensus.cc:2454] T 2a57d4e298464ad580027819241d4735 P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb3125d57f2741698f075887542fc1fc in term 0.
I20240502 13:59:40.554317 1418 raft_consensus.cc:2454] T 2a57d4e298464ad580027819241d4735 P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb3125d57f2741698f075887542fc1fc in term 0.
I20240502 13:59:40.554531 1229 leader_election.cc:302] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8feeac9977814b8b901607ec4071efdf, fb3125d57f2741698f075887542fc1fc; no voters:
I20240502 13:59:40.554672 1471 raft_consensus.cc:2781] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:40.554726 1471 raft_consensus.cc:482] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:40.554769 1471 raft_consensus.cc:3037] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.555364 1471 raft_consensus.cc:504] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } } peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } }
I20240502 13:59:40.555655 1471 leader_election.cc:288] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc [CANDIDATE]: Term 1 election: Requested vote from peers 8feeac9977814b8b901607ec4071efdf (127.0.145.66:45147), 0c0b5fc6c9894f7ca9a083e986452dff (127.0.145.67:41377)
I20240502 13:59:40.555888 1418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a57d4e298464ad580027819241d4735" candidate_uuid: "fb3125d57f2741698f075887542fc1fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c0b5fc6c9894f7ca9a083e986452dff"
I20240502 13:59:40.555895 1340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a57d4e298464ad580027819241d4735" candidate_uuid: "fb3125d57f2741698f075887542fc1fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8feeac9977814b8b901607ec4071efdf"
I20240502 13:59:40.555984 1418 raft_consensus.cc:3037] T 2a57d4e298464ad580027819241d4735 P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.556001 1340 raft_consensus.cc:3037] T 2a57d4e298464ad580027819241d4735 P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.556533 1418 raft_consensus.cc:2454] T 2a57d4e298464ad580027819241d4735 P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb3125d57f2741698f075887542fc1fc in term 1.
I20240502 13:59:40.556535 1340 raft_consensus.cc:2454] T 2a57d4e298464ad580027819241d4735 P 8feeac9977814b8b901607ec4071efdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb3125d57f2741698f075887542fc1fc in term 1.
I20240502 13:59:40.556811 1227 leader_election.cc:302] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c0b5fc6c9894f7ca9a083e986452dff, fb3125d57f2741698f075887542fc1fc; no voters:
I20240502 13:59:40.557014 1471 raft_consensus.cc:2781] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:40.557085 1471 raft_consensus.cc:686] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc [term 1 LEADER]: Becoming Leader. State: Replica: fb3125d57f2741698f075887542fc1fc, State: Running, Role: LEADER
I20240502 13:59:40.557194 1471 consensus_queue.cc:237] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } } peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } }
I20240502 13:59:40.558358 1158 catalog_manager.cc:5484] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc reported cstate change: term changed from 0 to 1, leader changed from <none> to fb3125d57f2741698f075887542fc1fc (127.0.145.65). New cstate: current_term: 1 leader_uuid: "fb3125d57f2741698f075887542fc1fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } health_report { overall_health: HEALTHY } } }
I20240502 13:59:40.571516 1477 raft_consensus.cc:482] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:40.571586 1477 raft_consensus.cc:504] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.571847 1477 leader_election.cc:288] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 8feeac9977814b8b901607ec4071efdf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb3125d57f2741698f075887542fc1fc (127.0.145.65:34005), 0c0b5fc6c9894f7ca9a083e986452dff (127.0.145.67:41377)
I20240502 13:59:40.572079 1264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1be05bf52a4de19cc2691aa7f73aa6" candidate_uuid: "8feeac9977814b8b901607ec4071efdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb3125d57f2741698f075887542fc1fc" is_pre_election: true
I20240502 13:59:40.572188 1264 raft_consensus.cc:2454] T 9f1be05bf52a4de19cc2691aa7f73aa6 P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8feeac9977814b8b901607ec4071efdf in term 0.
I20240502 13:59:40.572193 1418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1be05bf52a4de19cc2691aa7f73aa6" candidate_uuid: "8feeac9977814b8b901607ec4071efdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" is_pre_election: true
I20240502 13:59:40.572402 1305 leader_election.cc:302] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 8feeac9977814b8b901607ec4071efdf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8feeac9977814b8b901607ec4071efdf, fb3125d57f2741698f075887542fc1fc; no voters:
I20240502 13:59:40.572476 1418 raft_consensus.cc:2454] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8feeac9977814b8b901607ec4071efdf in term 0.
I20240502 13:59:40.572544 1477 raft_consensus.cc:2781] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:40.572607 1477 raft_consensus.cc:482] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:40.572651 1477 raft_consensus.cc:3037] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.573266 1477 raft_consensus.cc:504] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 8feeac9977814b8b901607ec4071efdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.573522 1477 leader_election.cc:288] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 8feeac9977814b8b901607ec4071efdf [CANDIDATE]: Term 1 election: Requested vote from peers fb3125d57f2741698f075887542fc1fc (127.0.145.65:34005), 0c0b5fc6c9894f7ca9a083e986452dff (127.0.145.67:41377)
I20240502 13:59:40.573745 1418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1be05bf52a4de19cc2691aa7f73aa6" candidate_uuid: "8feeac9977814b8b901607ec4071efdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c0b5fc6c9894f7ca9a083e986452dff"
I20240502 13:59:40.573784 1264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1be05bf52a4de19cc2691aa7f73aa6" candidate_uuid: "8feeac9977814b8b901607ec4071efdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb3125d57f2741698f075887542fc1fc"
I20240502 13:59:40.573832 1418 raft_consensus.cc:3037] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.573859 1264 raft_consensus.cc:3037] T 9f1be05bf52a4de19cc2691aa7f73aa6 P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.574448 1418 raft_consensus.cc:2454] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8feeac9977814b8b901607ec4071efdf in term 1.
I20240502 13:59:40.574496 1264 raft_consensus.cc:2454] T 9f1be05bf52a4de19cc2691aa7f73aa6 P fb3125d57f2741698f075887542fc1fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8feeac9977814b8b901607ec4071efdf in term 1.
I20240502 13:59:40.574708 1305 leader_election.cc:302] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 8feeac9977814b8b901607ec4071efdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8feeac9977814b8b901607ec4071efdf, fb3125d57f2741698f075887542fc1fc; no voters:
I20240502 13:59:40.574851 1477 raft_consensus.cc:2781] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 8feeac9977814b8b901607ec4071efdf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:40.574928 1477 raft_consensus.cc:686] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 8feeac9977814b8b901607ec4071efdf [term 1 LEADER]: Becoming Leader. State: Replica: 8feeac9977814b8b901607ec4071efdf, State: Running, Role: LEADER
I20240502 13:59:40.575057 1477 consensus_queue.cc:237] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 8feeac9977814b8b901607ec4071efdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.576284 1158 catalog_manager.cc:5484] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 8feeac9977814b8b901607ec4071efdf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8feeac9977814b8b901607ec4071efdf (127.0.145.66). New cstate: current_term: 1 leader_uuid: "8feeac9977814b8b901607ec4071efdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:40.594900 1476 raft_consensus.cc:482] T 7a8684e4da8f46b6acace6ccac46991f P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:40.594991 1476 raft_consensus.cc:504] T 7a8684e4da8f46b6acace6ccac46991f P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.595264 1476 leader_election.cc:288] T 7a8684e4da8f46b6acace6ccac46991f P 0c0b5fc6c9894f7ca9a083e986452dff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb3125d57f2741698f075887542fc1fc (127.0.145.65:34005), 8feeac9977814b8b901607ec4071efdf (127.0.145.66:45147)
I20240502 13:59:40.595561 1264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a8684e4da8f46b6acace6ccac46991f" candidate_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb3125d57f2741698f075887542fc1fc" is_pre_election: true
I20240502 13:59:40.595561 1340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a8684e4da8f46b6acace6ccac46991f" candidate_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8feeac9977814b8b901607ec4071efdf" is_pre_election: true
I20240502 13:59:40.595702 1264 raft_consensus.cc:2454] T 7a8684e4da8f46b6acace6ccac46991f P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c0b5fc6c9894f7ca9a083e986452dff in term 0.
I20240502 13:59:40.595703 1340 raft_consensus.cc:2454] T 7a8684e4da8f46b6acace6ccac46991f P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c0b5fc6c9894f7ca9a083e986452dff in term 0.
I20240502 13:59:40.595949 1382 leader_election.cc:302] T 7a8684e4da8f46b6acace6ccac46991f P 0c0b5fc6c9894f7ca9a083e986452dff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c0b5fc6c9894f7ca9a083e986452dff, 8feeac9977814b8b901607ec4071efdf; no voters:
I20240502 13:59:40.596073 1476 raft_consensus.cc:2781] T 7a8684e4da8f46b6acace6ccac46991f P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:40.596129 1476 raft_consensus.cc:482] T 7a8684e4da8f46b6acace6ccac46991f P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:40.596154 1476 raft_consensus.cc:3037] T 7a8684e4da8f46b6acace6ccac46991f P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.596846 1476 raft_consensus.cc:504] T 7a8684e4da8f46b6acace6ccac46991f P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.597105 1476 leader_election.cc:288] T 7a8684e4da8f46b6acace6ccac46991f P 0c0b5fc6c9894f7ca9a083e986452dff [CANDIDATE]: Term 1 election: Requested vote from peers fb3125d57f2741698f075887542fc1fc (127.0.145.65:34005), 8feeac9977814b8b901607ec4071efdf (127.0.145.66:45147)
I20240502 13:59:40.597430 1340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a8684e4da8f46b6acace6ccac46991f" candidate_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8feeac9977814b8b901607ec4071efdf"
I20240502 13:59:40.597430 1264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a8684e4da8f46b6acace6ccac46991f" candidate_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb3125d57f2741698f075887542fc1fc"
I20240502 13:59:40.597576 1340 raft_consensus.cc:3037] T 7a8684e4da8f46b6acace6ccac46991f P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.597584 1264 raft_consensus.cc:3037] T 7a8684e4da8f46b6acace6ccac46991f P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.598364 1264 raft_consensus.cc:2454] T 7a8684e4da8f46b6acace6ccac46991f P fb3125d57f2741698f075887542fc1fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c0b5fc6c9894f7ca9a083e986452dff in term 1.
I20240502 13:59:40.598364 1340 raft_consensus.cc:2454] T 7a8684e4da8f46b6acace6ccac46991f P 8feeac9977814b8b901607ec4071efdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c0b5fc6c9894f7ca9a083e986452dff in term 1.
I20240502 13:59:40.598654 1382 leader_election.cc:302] T 7a8684e4da8f46b6acace6ccac46991f P 0c0b5fc6c9894f7ca9a083e986452dff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c0b5fc6c9894f7ca9a083e986452dff, fb3125d57f2741698f075887542fc1fc; no voters:
I20240502 13:59:40.598798 1476 raft_consensus.cc:2781] T 7a8684e4da8f46b6acace6ccac46991f P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:40.598974 1476 raft_consensus.cc:686] T 7a8684e4da8f46b6acace6ccac46991f P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 LEADER]: Becoming Leader. State: Replica: 0c0b5fc6c9894f7ca9a083e986452dff, State: Running, Role: LEADER
I20240502 13:59:40.599109 1476 consensus_queue.cc:237] T 7a8684e4da8f46b6acace6ccac46991f P 0c0b5fc6c9894f7ca9a083e986452dff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.600396 1158 catalog_manager.cc:5484] T 7a8684e4da8f46b6acace6ccac46991f P 0c0b5fc6c9894f7ca9a083e986452dff reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c0b5fc6c9894f7ca9a083e986452dff (127.0.145.67). New cstate: current_term: 1 leader_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } health_report { overall_health: HEALTHY } } }
I20240502 13:59:40.697075 1471 raft_consensus.cc:482] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:40.697194 1471 raft_consensus.cc:504] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.697547 1471 leader_election.cc:288] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8feeac9977814b8b901607ec4071efdf (127.0.145.66:45147), 0c0b5fc6c9894f7ca9a083e986452dff (127.0.145.67:41377)
I20240502 13:59:40.697890 1340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b5648798faa4b2997198f954af3d4e8" candidate_uuid: "fb3125d57f2741698f075887542fc1fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8feeac9977814b8b901607ec4071efdf" is_pre_election: true
I20240502 13:59:40.697890 1418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b5648798faa4b2997198f954af3d4e8" candidate_uuid: "fb3125d57f2741698f075887542fc1fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" is_pre_election: true
I20240502 13:59:40.698067 1340 raft_consensus.cc:2454] T 3b5648798faa4b2997198f954af3d4e8 P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb3125d57f2741698f075887542fc1fc in term 0.
I20240502 13:59:40.698071 1418 raft_consensus.cc:2454] T 3b5648798faa4b2997198f954af3d4e8 P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb3125d57f2741698f075887542fc1fc in term 0.
I20240502 13:59:40.698345 1229 leader_election.cc:302] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8feeac9977814b8b901607ec4071efdf, fb3125d57f2741698f075887542fc1fc; no voters:
I20240502 13:59:40.698505 1471 raft_consensus.cc:2781] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:40.698559 1471 raft_consensus.cc:482] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:40.698585 1471 raft_consensus.cc:3037] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.699293 1471 raft_consensus.cc:504] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.699585 1471 leader_election.cc:288] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc [CANDIDATE]: Term 1 election: Requested vote from peers 8feeac9977814b8b901607ec4071efdf (127.0.145.66:45147), 0c0b5fc6c9894f7ca9a083e986452dff (127.0.145.67:41377)
I20240502 13:59:40.699855 1418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b5648798faa4b2997198f954af3d4e8" candidate_uuid: "fb3125d57f2741698f075887542fc1fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c0b5fc6c9894f7ca9a083e986452dff"
I20240502 13:59:40.699855 1340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b5648798faa4b2997198f954af3d4e8" candidate_uuid: "fb3125d57f2741698f075887542fc1fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8feeac9977814b8b901607ec4071efdf"
I20240502 13:59:40.699997 1340 raft_consensus.cc:3037] T 3b5648798faa4b2997198f954af3d4e8 P 8feeac9977814b8b901607ec4071efdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.699996 1418 raft_consensus.cc:3037] T 3b5648798faa4b2997198f954af3d4e8 P 0c0b5fc6c9894f7ca9a083e986452dff [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.700855 1418 raft_consensus.cc:2454] T 3b5648798faa4b2997198f954af3d4e8 P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb3125d57f2741698f075887542fc1fc in term 1.
I20240502 13:59:40.700855 1340 raft_consensus.cc:2454] T 3b5648798faa4b2997198f954af3d4e8 P 8feeac9977814b8b901607ec4071efdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb3125d57f2741698f075887542fc1fc in term 1.
I20240502 13:59:40.701125 1227 leader_election.cc:302] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c0b5fc6c9894f7ca9a083e986452dff, fb3125d57f2741698f075887542fc1fc; no voters:
I20240502 13:59:40.701272 1471 raft_consensus.cc:2781] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:40.701375 1471 raft_consensus.cc:686] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc [term 1 LEADER]: Becoming Leader. State: Replica: fb3125d57f2741698f075887542fc1fc, State: Running, Role: LEADER
I20240502 13:59:40.701501 1471 consensus_queue.cc:237] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:40.702888 1158 catalog_manager.cc:5484] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc reported cstate change: term changed from 0 to 1, leader changed from <none> to fb3125d57f2741698f075887542fc1fc (127.0.145.65). New cstate: current_term: 1 leader_uuid: "fb3125d57f2741698f075887542fc1fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:40.744311 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:40.745970 1512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:40.746426 1515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:40.746809 1513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:40.747179 581 server_base.cc:1034] running on GCE node
I20240502 13:59:40.747321 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 13:59:40.747356 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 13:59:40.747370 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658380747370 us; error 0 us; skew 500 ppm
I20240502 13:59:40.748291 581 webserver.cc:460] Webserver started at http://127.0.145.68:42095/ using document root <none> and password file <none>
I20240502 13:59:40.748420 581 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:40.748467 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:40.748533 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:40.748894 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658376531600-581-0/minicluster-data/ts-3-root/instance:
uuid: "60c7ffc01d7844d4a6744257ed253486"
format_stamp: "Formatted at 2024-05-02 13:59:40 on dist-test-slave-1z5j"
I20240502 13:59:40.750154 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:40.750983 1520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:40.751201 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:40.751266 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658376531600-581-0/minicluster-data/ts-3-root
uuid: "60c7ffc01d7844d4a6744257ed253486"
format_stamp: "Formatted at 2024-05-02 13:59:40 on dist-test-slave-1z5j"
I20240502 13:59:40.751323 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658376531600-581-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658376531600-581-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658376531600-581-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:40.757139 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:40.757403 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:40.757643 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:40.758013 581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:40.758049 581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:40.758086 581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:40.758113 581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:40.762814 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.68:46387
I20240502 13:59:40.762852 1583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.68:46387 every 8 connection(s)
I20240502 13:59:40.766794 1584 heartbeater.cc:347] Connected to a master server at 127.0.145.126:41703
I20240502 13:59:40.766868 1584 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:40.767021 1584 heartbeater.cc:511] Master 127.0.145.126:41703 requested a full tablet report, sending...
I20240502 13:59:40.767482 1158 ts_manager.cc:194] Registered new tserver with Master: 60c7ffc01d7844d4a6744257ed253486 (127.0.145.68:46387)
I20240502 13:59:40.767917 1158 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41658
I20240502 13:59:40.925686 1471 consensus_queue.cc:1035] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:40.928697 1589 consensus_queue.cc:1035] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:40.975173 1589 consensus_queue.cc:1035] T fb8f611a150140b4827890f46869b33f P fb3125d57f2741698f075887542fc1fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:40.976258 1589 consensus_queue.cc:1035] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:40.977639 1471 consensus_queue.cc:1035] T fb8f611a150140b4827890f46869b33f P fb3125d57f2741698f075887542fc1fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:40.980542 1469 consensus_queue.cc:1035] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:40.984534 1477 consensus_queue.cc:1035] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:40.986848 1483 consensus_queue.cc:1035] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:41.075604 1483 consensus_queue.cc:1035] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 8feeac9977814b8b901607ec4071efdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:41.078058 1598 consensus_queue.cc:1035] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 8feeac9977814b8b901607ec4071efdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:41.120095 1469 consensus_queue.cc:1035] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:41.122426 1471 consensus_queue.cc:1035] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:41.141793 1509 consensus_queue.cc:1035] T 7a8684e4da8f46b6acace6ccac46991f P 0c0b5fc6c9894f7ca9a083e986452dff [LEADER]: Connected to new peer: Peer: permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:41.143990 1605 consensus_queue.cc:1035] T 7a8684e4da8f46b6acace6ccac46991f P 0c0b5fc6c9894f7ca9a083e986452dff [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:41.181010 1471 consensus_queue.cc:1035] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:41.183313 1471 consensus_queue.cc:1035] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:41.202569 1215 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 13:59:41.207304 1264 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2a57d4e298464ad580027819241d4735"
dest_uuid: "fb3125d57f2741698f075887542fc1fc"
mode: GRACEFUL
new_leader_uuid: "0c0b5fc6c9894f7ca9a083e986452dff"
from {username='slave'} at 127.0.0.1:34586
I20240502 13:59:41.207454 1264 raft_consensus.cc:595] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc [term 1 LEADER]: Received request to transfer leadership to TS 0c0b5fc6c9894f7ca9a083e986452dff
I20240502 13:59:41.208120 1264 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2cad5291c5eb4f12a6bbcbcfdb37a0ed"
dest_uuid: "fb3125d57f2741698f075887542fc1fc"
mode: GRACEFUL
new_leader_uuid: "8feeac9977814b8b901607ec4071efdf"
from {username='slave'} at 127.0.0.1:34586
I20240502 13:59:41.208215 1264 raft_consensus.cc:595] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc [term 1 LEADER]: Received request to transfer leadership to TS 8feeac9977814b8b901607ec4071efdf
I20240502 13:59:41.208473 1215 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240502 13:59:41.208647 1215 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 13:59:41.439220 1469 raft_consensus.cc:979] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc: : Instructing follower 8feeac9977814b8b901607ec4071efdf to start an election
I20240502 13:59:41.439360 1469 raft_consensus.cc:1067] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc [term 1 LEADER]: Signalling peer 8feeac9977814b8b901607ec4071efdf to start an election
I20240502 13:59:41.439949 1340 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2cad5291c5eb4f12a6bbcbcfdb37a0ed"
dest_uuid: "8feeac9977814b8b901607ec4071efdf"
from {username='slave'} at 127.0.0.1:54862
I20240502 13:59:41.440097 1340 raft_consensus.cc:482] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 8feeac9977814b8b901607ec4071efdf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 13:59:41.440163 1340 raft_consensus.cc:3037] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 8feeac9977814b8b901607ec4071efdf [term 1 FOLLOWER]: Advancing to term 2
I20240502 13:59:41.440984 1340 raft_consensus.cc:504] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 8feeac9977814b8b901607ec4071efdf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:41.441344 1340 leader_election.cc:288] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 8feeac9977814b8b901607ec4071efdf [CANDIDATE]: Term 2 election: Requested vote from peers fb3125d57f2741698f075887542fc1fc (127.0.145.65:34005), 0c0b5fc6c9894f7ca9a083e986452dff (127.0.145.67:41377)
I20240502 13:59:41.441571 1264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cad5291c5eb4f12a6bbcbcfdb37a0ed" candidate_uuid: "8feeac9977814b8b901607ec4071efdf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fb3125d57f2741698f075887542fc1fc"
I20240502 13:59:41.441642 1417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cad5291c5eb4f12a6bbcbcfdb37a0ed" candidate_uuid: "8feeac9977814b8b901607ec4071efdf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0c0b5fc6c9894f7ca9a083e986452dff"
I20240502 13:59:41.441687 1264 raft_consensus.cc:3032] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc [term 1 LEADER]: Stepping down as leader of term 1
I20240502 13:59:41.441785 1417 raft_consensus.cc:3037] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 FOLLOWER]: Advancing to term 2
I20240502 13:59:41.441785 1264 raft_consensus.cc:729] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb3125d57f2741698f075887542fc1fc, State: Running, Role: LEADER
I20240502 13:59:41.441962 1264 consensus_queue.cc:260] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:41.442107 1264 raft_consensus.cc:3037] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc [term 1 FOLLOWER]: Advancing to term 2
I20240502 13:59:41.442621 1417 raft_consensus.cc:2454] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 0c0b5fc6c9894f7ca9a083e986452dff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8feeac9977814b8b901607ec4071efdf in term 2.
I20240502 13:59:41.442761 1264 raft_consensus.cc:2454] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8feeac9977814b8b901607ec4071efdf in term 2.
I20240502 13:59:41.442858 1303 leader_election.cc:302] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 8feeac9977814b8b901607ec4071efdf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c0b5fc6c9894f7ca9a083e986452dff, 8feeac9977814b8b901607ec4071efdf; no voters:
I20240502 13:59:41.443020 1598 raft_consensus.cc:2781] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 8feeac9977814b8b901607ec4071efdf [term 2 FOLLOWER]: Leader election won for term 2
I20240502 13:59:41.443087 1598 raft_consensus.cc:686] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 8feeac9977814b8b901607ec4071efdf [term 2 LEADER]: Becoming Leader. State: Replica: 8feeac9977814b8b901607ec4071efdf, State: Running, Role: LEADER
I20240502 13:59:41.443205 1598 consensus_queue.cc:237] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 8feeac9977814b8b901607ec4071efdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } }
I20240502 13:59:41.444478 1158 catalog_manager.cc:5484] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 8feeac9977814b8b901607ec4071efdf reported cstate change: term changed from 1 to 2, leader changed from fb3125d57f2741698f075887542fc1fc (127.0.145.65) to 8feeac9977814b8b901607ec4071efdf (127.0.145.66). New cstate: current_term: 2 leader_uuid: "8feeac9977814b8b901607ec4071efdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:41.525822 1589 raft_consensus.cc:979] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc: : Instructing follower 0c0b5fc6c9894f7ca9a083e986452dff to start an election
I20240502 13:59:41.525954 1471 raft_consensus.cc:1067] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc [term 1 LEADER]: Signalling peer 0c0b5fc6c9894f7ca9a083e986452dff to start an election
I20240502 13:59:41.526348 1417 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2a57d4e298464ad580027819241d4735"
dest_uuid: "0c0b5fc6c9894f7ca9a083e986452dff"
from {username='slave'} at 127.0.0.1:46534
I20240502 13:59:41.526474 1417 raft_consensus.cc:482] T 2a57d4e298464ad580027819241d4735 P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 13:59:41.526542 1417 raft_consensus.cc:3037] T 2a57d4e298464ad580027819241d4735 P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 FOLLOWER]: Advancing to term 2
I20240502 13:59:41.527326 1417 raft_consensus.cc:504] T 2a57d4e298464ad580027819241d4735 P 0c0b5fc6c9894f7ca9a083e986452dff [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } } peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } }
I20240502 13:59:41.527657 1417 leader_election.cc:288] T 2a57d4e298464ad580027819241d4735 P 0c0b5fc6c9894f7ca9a083e986452dff [CANDIDATE]: Term 2 election: Requested vote from peers 8feeac9977814b8b901607ec4071efdf (127.0.145.66:45147), fb3125d57f2741698f075887542fc1fc (127.0.145.65:34005)
I20240502 13:59:41.527902 1340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a57d4e298464ad580027819241d4735" candidate_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8feeac9977814b8b901607ec4071efdf"
I20240502 13:59:41.528026 1340 raft_consensus.cc:3037] T 2a57d4e298464ad580027819241d4735 P 8feeac9977814b8b901607ec4071efdf [term 1 FOLLOWER]: Advancing to term 2
I20240502 13:59:41.527990 1264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a57d4e298464ad580027819241d4735" candidate_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fb3125d57f2741698f075887542fc1fc"
I20240502 13:59:41.528126 1264 raft_consensus.cc:3032] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc [term 1 LEADER]: Stepping down as leader of term 1
I20240502 13:59:41.528162 1264 raft_consensus.cc:729] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb3125d57f2741698f075887542fc1fc, State: Running, Role: LEADER
I20240502 13:59:41.528292 1264 consensus_queue.cc:260] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } } peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } }
I20240502 13:59:41.528430 1264 raft_consensus.cc:3037] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc [term 1 FOLLOWER]: Advancing to term 2
I20240502 13:59:41.528755 1340 raft_consensus.cc:2454] T 2a57d4e298464ad580027819241d4735 P 8feeac9977814b8b901607ec4071efdf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c0b5fc6c9894f7ca9a083e986452dff in term 2.
I20240502 13:59:41.529048 1382 leader_election.cc:302] T 2a57d4e298464ad580027819241d4735 P 0c0b5fc6c9894f7ca9a083e986452dff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c0b5fc6c9894f7ca9a083e986452dff, 8feeac9977814b8b901607ec4071efdf; no voters:
I20240502 13:59:41.529167 1264 raft_consensus.cc:2454] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c0b5fc6c9894f7ca9a083e986452dff in term 2.
I20240502 13:59:41.529193 1605 raft_consensus.cc:2781] T 2a57d4e298464ad580027819241d4735 P 0c0b5fc6c9894f7ca9a083e986452dff [term 2 FOLLOWER]: Leader election won for term 2
I20240502 13:59:41.529280 1605 raft_consensus.cc:686] T 2a57d4e298464ad580027819241d4735 P 0c0b5fc6c9894f7ca9a083e986452dff [term 2 LEADER]: Becoming Leader. State: Replica: 0c0b5fc6c9894f7ca9a083e986452dff, State: Running, Role: LEADER
I20240502 13:59:41.529395 1605 consensus_queue.cc:237] T 2a57d4e298464ad580027819241d4735 P 0c0b5fc6c9894f7ca9a083e986452dff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } } peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } }
I20240502 13:59:41.530606 1158 catalog_manager.cc:5484] T 2a57d4e298464ad580027819241d4735 P 0c0b5fc6c9894f7ca9a083e986452dff reported cstate change: term changed from 1 to 2, leader changed from fb3125d57f2741698f075887542fc1fc (127.0.145.65) to 0c0b5fc6c9894f7ca9a083e986452dff (127.0.145.67). New cstate: current_term: 2 leader_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:41.768642 1584 heartbeater.cc:503] Master 127.0.145.126:41703 was elected leader, sending a full tablet report...
I20240502 13:59:41.843441 1264 raft_consensus.cc:1261] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc [term 2 FOLLOWER]: Refusing update from remote peer 8feeac9977814b8b901607ec4071efdf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 13:59:41.843814 1598 consensus_queue.cc:1035] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 8feeac9977814b8b901607ec4071efdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 13:59:41.845798 1417 raft_consensus.cc:1261] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 0c0b5fc6c9894f7ca9a083e986452dff [term 2 FOLLOWER]: Refusing update from remote peer 8feeac9977814b8b901607ec4071efdf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 13:59:41.846104 1598 consensus_queue.cc:1035] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 8feeac9977814b8b901607ec4071efdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 13:59:42.016683 1264 raft_consensus.cc:1261] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc [term 2 FOLLOWER]: Refusing update from remote peer 0c0b5fc6c9894f7ca9a083e986452dff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 13:59:42.017141 1615 consensus_queue.cc:1035] T 2a57d4e298464ad580027819241d4735 P 0c0b5fc6c9894f7ca9a083e986452dff [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 13:59:42.019306 1340 raft_consensus.cc:1261] T 2a57d4e298464ad580027819241d4735 P 8feeac9977814b8b901607ec4071efdf [term 2 FOLLOWER]: Refusing update from remote peer 0c0b5fc6c9894f7ca9a083e986452dff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 13:59:42.019644 1605 consensus_queue.cc:1035] T 2a57d4e298464ad580027819241d4735 P 0c0b5fc6c9894f7ca9a083e986452dff [LEADER]: Connected to new peer: Peer: permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 13:59:43.208914 1215 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 13:59:43.209697 1215 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 13:59:43.209780 1215 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 13:59:45.210012 1215 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 13:59:45.210783 1215 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 13:59:45.210870 1215 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 13:59:46.215636 1340 consensus_queue.cc:237] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 8feeac9977814b8b901607ec4071efdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } attrs { replace: true } } peers { permanent_uuid: "60c7ffc01d7844d4a6744257ed253486" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 46387 } attrs { promote: true } }
I20240502 13:59:46.217358 1264 raft_consensus.cc:1261] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc [term 2 FOLLOWER]: Refusing update from remote peer 8feeac9977814b8b901607ec4071efdf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 13:59:46.217510 1418 raft_consensus.cc:1261] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 0c0b5fc6c9894f7ca9a083e986452dff [term 2 FOLLOWER]: Refusing update from remote peer 8feeac9977814b8b901607ec4071efdf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 13:59:46.217756 1622 consensus_queue.cc:1035] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 8feeac9977814b8b901607ec4071efdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 13:59:46.217960 1628 consensus_queue.cc:1035] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 8feeac9977814b8b901607ec4071efdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 13:59:46.219576 1627 raft_consensus.cc:2932] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 8feeac9977814b8b901607ec4071efdf [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 60c7ffc01d7844d4a6744257ed253486 (127.0.145.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } attrs { replace: true } } peers { permanent_uuid: "60c7ffc01d7844d4a6744257ed253486" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 46387 } attrs { promote: true } } }
I20240502 13:59:46.219902 1264 raft_consensus.cc:2932] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 60c7ffc01d7844d4a6744257ed253486 (127.0.145.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } attrs { replace: true } } peers { permanent_uuid: "60c7ffc01d7844d4a6744257ed253486" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 46387 } attrs { promote: true } } }
W20240502 13:59:46.221470 1306 consensus_peers.cc:485] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 8feeac9977814b8b901607ec4071efdf -> Peer 60c7ffc01d7844d4a6744257ed253486 (127.0.145.68:46387): Couldn't send request to peer 60c7ffc01d7844d4a6744257ed253486. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2cad5291c5eb4f12a6bbcbcfdb37a0ed. This is attempt 1: this message will repeat every 5th retry.
I20240502 13:59:46.222086 1158 catalog_manager.cc:5484] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc reported cstate change: config changed from index -1 to 3, NON_VOTER 60c7ffc01d7844d4a6744257ed253486 (127.0.145.68) added. New cstate: current_term: 2 leader_uuid: "8feeac9977814b8b901607ec4071efdf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } attrs { replace: true } } peers { permanent_uuid: "60c7ffc01d7844d4a6744257ed253486" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 46387 } attrs { promote: true } } }
I20240502 13:59:46.222240 1418 raft_consensus.cc:2932] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 0c0b5fc6c9894f7ca9a083e986452dff [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 60c7ffc01d7844d4a6744257ed253486 (127.0.145.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } attrs { replace: true } } peers { permanent_uuid: "60c7ffc01d7844d4a6744257ed253486" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 46387 } attrs { promote: true } } }
I20240502 13:59:46.222613 1340 consensus_queue.cc:237] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } attrs { replace: true } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } } peers { permanent_uuid: "60c7ffc01d7844d4a6744257ed253486" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 46387 } attrs { promote: true } }
I20240502 13:59:46.224084 1418 raft_consensus.cc:1261] T 55b1c1f11b1449d8954ef0c95a95b6da P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 FOLLOWER]: Refusing update from remote peer 8feeac9977814b8b901607ec4071efdf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 13:59:46.224372 1306 consensus_peers.cc:485] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf -> Peer 60c7ffc01d7844d4a6744257ed253486 (127.0.145.68:46387): Couldn't send request to peer 60c7ffc01d7844d4a6744257ed253486. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 55b1c1f11b1449d8954ef0c95a95b6da. This is attempt 1: this message will repeat every 5th retry.
I20240502 13:59:46.224401 1622 consensus_queue.cc:1035] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 13:59:46.224501 1264 raft_consensus.cc:1261] T 55b1c1f11b1449d8954ef0c95a95b6da P fb3125d57f2741698f075887542fc1fc [term 1 FOLLOWER]: Refusing update from remote peer 8feeac9977814b8b901607ec4071efdf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:46.224857 1622 consensus_queue.cc:1035] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 13:59:46.226016 1636 raft_consensus.cc:2932] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 60c7ffc01d7844d4a6744257ed253486 (127.0.145.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } attrs { replace: true } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } } peers { permanent_uuid: "60c7ffc01d7844d4a6744257ed253486" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 46387 } attrs { promote: true } } }
I20240502 13:59:46.226343 1418 raft_consensus.cc:2932] T 55b1c1f11b1449d8954ef0c95a95b6da P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 60c7ffc01d7844d4a6744257ed253486 (127.0.145.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } attrs { replace: true } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } } peers { permanent_uuid: "60c7ffc01d7844d4a6744257ed253486" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 46387 } attrs { promote: true } } }
I20240502 13:59:46.227002 1264 raft_consensus.cc:2932] T 55b1c1f11b1449d8954ef0c95a95b6da P fb3125d57f2741698f075887542fc1fc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 60c7ffc01d7844d4a6744257ed253486 (127.0.145.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } attrs { replace: true } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } } peers { permanent_uuid: "60c7ffc01d7844d4a6744257ed253486" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 46387 } attrs { promote: true } } }
I20240502 13:59:46.228214 1161 catalog_manager.cc:5484] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf reported cstate change: config changed from index -1 to 2, NON_VOTER 60c7ffc01d7844d4a6744257ed253486 (127.0.145.68) added. New cstate: current_term: 1 leader_uuid: "8feeac9977814b8b901607ec4071efdf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60c7ffc01d7844d4a6744257ed253486" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 46387 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:46.231778 1264 consensus_queue.cc:237] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } } peers { permanent_uuid: "60c7ffc01d7844d4a6744257ed253486" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 46387 } attrs { promote: true } }
I20240502 13:59:46.233065 1340 raft_consensus.cc:1261] T 3b5648798faa4b2997198f954af3d4e8 P 8feeac9977814b8b901607ec4071efdf [term 1 FOLLOWER]: Refusing update from remote peer fb3125d57f2741698f075887542fc1fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:46.233116 1418 raft_consensus.cc:1261] T 3b5648798faa4b2997198f954af3d4e8 P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 FOLLOWER]: Refusing update from remote peer fb3125d57f2741698f075887542fc1fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:46.233433 1644 consensus_queue.cc:1035] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 13:59:46.233594 1589 consensus_queue.cc:1035] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 13:59:46.234972 1589 raft_consensus.cc:2932] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 60c7ffc01d7844d4a6744257ed253486 (127.0.145.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } } peers { permanent_uuid: "60c7ffc01d7844d4a6744257ed253486" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 46387 } attrs { promote: true } } }
W20240502 13:59:46.236277 1230 consensus_peers.cc:485] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc -> Peer 60c7ffc01d7844d4a6744257ed253486 (127.0.145.68:46387): Couldn't send request to peer 60c7ffc01d7844d4a6744257ed253486. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b5648798faa4b2997198f954af3d4e8. This is attempt 1: this message will repeat every 5th retry.
I20240502 13:59:46.236212 1418 raft_consensus.cc:2932] T 3b5648798faa4b2997198f954af3d4e8 P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 60c7ffc01d7844d4a6744257ed253486 (127.0.145.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } } peers { permanent_uuid: "60c7ffc01d7844d4a6744257ed253486" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 46387 } attrs { promote: true } } }
I20240502 13:59:46.237013 1160 catalog_manager.cc:5484] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc reported cstate change: config changed from index -1 to 2, NON_VOTER 60c7ffc01d7844d4a6744257ed253486 (127.0.145.68) added. New cstate: current_term: 1 leader_uuid: "fb3125d57f2741698f075887542fc1fc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60c7ffc01d7844d4a6744257ed253486" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 46387 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:46.237212 1264 consensus_queue.cc:237] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } } peers { permanent_uuid: "60c7ffc01d7844d4a6744257ed253486" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 46387 } attrs { promote: true } }
I20240502 13:59:46.237830 1340 raft_consensus.cc:2932] T 3b5648798faa4b2997198f954af3d4e8 P 8feeac9977814b8b901607ec4071efdf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 60c7ffc01d7844d4a6744257ed253486 (127.0.145.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } } peers { permanent_uuid: "60c7ffc01d7844d4a6744257ed253486" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 46387 } attrs { promote: true } } }
W20240502 13:59:46.238708 1230 consensus_peers.cc:485] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc -> Peer 60c7ffc01d7844d4a6744257ed253486 (127.0.145.68:46387): Couldn't send request to peer 60c7ffc01d7844d4a6744257ed253486. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f9c42e0ae4dc4e4f96cbb9161babbbce. This is attempt 1: this message will repeat every 5th retry.
I20240502 13:59:46.238708 1339 raft_consensus.cc:1261] T f9c42e0ae4dc4e4f96cbb9161babbbce P 8feeac9977814b8b901607ec4071efdf [term 1 FOLLOWER]: Refusing update from remote peer fb3125d57f2741698f075887542fc1fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:46.239545 1418 raft_consensus.cc:1261] T f9c42e0ae4dc4e4f96cbb9161babbbce P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 FOLLOWER]: Refusing update from remote peer fb3125d57f2741698f075887542fc1fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:46.239853 1644 consensus_queue.cc:1035] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 13:59:46.240120 1649 consensus_queue.cc:1035] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 13:59:46.241488 1649 raft_consensus.cc:2932] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 60c7ffc01d7844d4a6744257ed253486 (127.0.145.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } } peers { permanent_uuid: "60c7ffc01d7844d4a6744257ed253486" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 46387 } attrs { promote: true } } }
I20240502 13:59:46.242170 1418 raft_consensus.cc:2932] T f9c42e0ae4dc4e4f96cbb9161babbbce P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 60c7ffc01d7844d4a6744257ed253486 (127.0.145.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } } peers { permanent_uuid: "60c7ffc01d7844d4a6744257ed253486" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 46387 } attrs { promote: true } } }
I20240502 13:59:46.242825 1339 raft_consensus.cc:2932] T f9c42e0ae4dc4e4f96cbb9161babbbce P 8feeac9977814b8b901607ec4071efdf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 60c7ffc01d7844d4a6744257ed253486 (127.0.145.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } } peers { permanent_uuid: "60c7ffc01d7844d4a6744257ed253486" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 46387 } attrs { promote: true } } }
I20240502 13:59:46.244529 1160 catalog_manager.cc:5484] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc reported cstate change: config changed from index -1 to 2, NON_VOTER 60c7ffc01d7844d4a6744257ed253486 (127.0.145.68) added. New cstate: current_term: 1 leader_uuid: "fb3125d57f2741698f075887542fc1fc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fb3125d57f2741698f075887542fc1fc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8feeac9977814b8b901607ec4071efdf" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 45147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c0b5fc6c9894f7ca9a083e986452dff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 41377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60c7ffc01d7844d4a6744257ed253486" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 46387 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:46.275851 581 tablet_server.cc:178] TabletServer@127.0.145.65:0 shutting down...
W20240502 13:59:46.277204 1146 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 1 similar messages]
W20240502 13:59:46.278896 1210 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 13:59:46.278956 1210 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 13:59:46.279564 581 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 13:59:46.279734 581 tablet_replica.cc:330] T fb8f611a150140b4827890f46869b33f P fb3125d57f2741698f075887542fc1fc: stopping tablet replica
I20240502 13:59:46.279862 581 raft_consensus.cc:2229] T fb8f611a150140b4827890f46869b33f P fb3125d57f2741698f075887542fc1fc [term 1 LEADER]: Raft consensus shutting down.
I20240502 13:59:46.280042 581 raft_consensus.cc:2258] T fb8f611a150140b4827890f46869b33f P fb3125d57f2741698f075887542fc1fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:46.280473 581 tablet_replica.cc:330] T 7a8684e4da8f46b6acace6ccac46991f P fb3125d57f2741698f075887542fc1fc: stopping tablet replica
I20240502 13:59:46.280572 581 raft_consensus.cc:2229] T 7a8684e4da8f46b6acace6ccac46991f P fb3125d57f2741698f075887542fc1fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 13:59:46.280654 581 raft_consensus.cc:2258] T 7a8684e4da8f46b6acace6ccac46991f P fb3125d57f2741698f075887542fc1fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:46.281028 581 tablet_replica.cc:330] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc: stopping tablet replica
I20240502 13:59:46.281111 581 raft_consensus.cc:2229] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 13:59:46.281190 581 raft_consensus.cc:2258] T 2a57d4e298464ad580027819241d4735 P fb3125d57f2741698f075887542fc1fc [term 2 FOLLOWER]: Raft consensus is shut down!
W20240502 13:59:46.281409 1210 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 13:59:46.281458 1210 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 13:59:46.281495 581 tablet_replica.cc:330] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc: stopping tablet replica
I20240502 13:59:46.281613 581 raft_consensus.cc:2229] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 13:59:46.281721 581 raft_consensus.cc:2258] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P fb3125d57f2741698f075887542fc1fc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:46.282135 581 tablet_replica.cc:330] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc: stopping tablet replica
I20240502 13:59:46.282254 581 raft_consensus.cc:2229] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc [term 1 LEADER]: Raft consensus shutting down.
I20240502 13:59:46.282447 581 raft_consensus.cc:2258] T f9c42e0ae4dc4e4f96cbb9161babbbce P fb3125d57f2741698f075887542fc1fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:46.282881 581 tablet_replica.cc:330] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc: stopping tablet replica
I20240502 13:59:46.282979 581 raft_consensus.cc:2229] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc [term 1 LEADER]: Raft consensus shutting down.
I20240502 13:59:46.283150 581 raft_consensus.cc:2258] T 3b5648798faa4b2997198f954af3d4e8 P fb3125d57f2741698f075887542fc1fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:46.283601 581 tablet_replica.cc:330] T 55b1c1f11b1449d8954ef0c95a95b6da P fb3125d57f2741698f075887542fc1fc: stopping tablet replica
I20240502 13:59:46.283690 581 raft_consensus.cc:2229] T 55b1c1f11b1449d8954ef0c95a95b6da P fb3125d57f2741698f075887542fc1fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 13:59:46.283779 581 raft_consensus.cc:2258] T 55b1c1f11b1449d8954ef0c95a95b6da P fb3125d57f2741698f075887542fc1fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:46.284185 581 tablet_replica.cc:330] T 9f1be05bf52a4de19cc2691aa7f73aa6 P fb3125d57f2741698f075887542fc1fc: stopping tablet replica
I20240502 13:59:46.284269 581 raft_consensus.cc:2229] T 9f1be05bf52a4de19cc2691aa7f73aa6 P fb3125d57f2741698f075887542fc1fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 13:59:46.284353 581 raft_consensus.cc:2258] T 9f1be05bf52a4de19cc2691aa7f73aa6 P fb3125d57f2741698f075887542fc1fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:46.298059 581 tablet_server.cc:194] TabletServer@127.0.145.65:0 shutdown complete.
I20240502 13:59:46.300319 581 tablet_server.cc:178] TabletServer@127.0.145.66:0 shutting down...
W20240502 13:59:46.301931 1210 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 13:59:46.302014 1210 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 13:59:46.303570 1210 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 13:59:46.303625 1210 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 13:59:46.304454 581 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 13:59:46.304613 581 tablet_replica.cc:330] T 2a57d4e298464ad580027819241d4735 P 8feeac9977814b8b901607ec4071efdf: stopping tablet replica
I20240502 13:59:46.304728 581 raft_consensus.cc:2229] T 2a57d4e298464ad580027819241d4735 P 8feeac9977814b8b901607ec4071efdf [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 13:59:46.304863 581 raft_consensus.cc:2258] T 2a57d4e298464ad580027819241d4735 P 8feeac9977814b8b901607ec4071efdf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:46.305176 581 tablet_replica.cc:330] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 8feeac9977814b8b901607ec4071efdf: stopping tablet replica
I20240502 13:59:46.305251 581 raft_consensus.cc:2229] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 8feeac9977814b8b901607ec4071efdf [term 2 LEADER]: Raft consensus shutting down.
I20240502 13:59:46.305389 581 raft_consensus.cc:2258] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 8feeac9977814b8b901607ec4071efdf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:46.305745 581 tablet_replica.cc:330] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf: stopping tablet replica
I20240502 13:59:46.305815 581 raft_consensus.cc:2229] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf [term 1 LEADER]: Raft consensus shutting down.
I20240502 13:59:46.305951 581 raft_consensus.cc:2258] T 55b1c1f11b1449d8954ef0c95a95b6da P 8feeac9977814b8b901607ec4071efdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:46.306340 581 tablet_replica.cc:330] T 3b5648798faa4b2997198f954af3d4e8 P 8feeac9977814b8b901607ec4071efdf: stopping tablet replica
I20240502 13:59:46.306411 581 raft_consensus.cc:2229] T 3b5648798faa4b2997198f954af3d4e8 P 8feeac9977814b8b901607ec4071efdf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 13:59:46.306489 581 raft_consensus.cc:2258] T 3b5648798faa4b2997198f954af3d4e8 P 8feeac9977814b8b901607ec4071efdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:46.306885 581 tablet_replica.cc:330] T 7a8684e4da8f46b6acace6ccac46991f P 8feeac9977814b8b901607ec4071efdf: stopping tablet replica
I20240502 13:59:46.306954 581 raft_consensus.cc:2229] T 7a8684e4da8f46b6acace6ccac46991f P 8feeac9977814b8b901607ec4071efdf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 13:59:46.307020 581 raft_consensus.cc:2258] T 7a8684e4da8f46b6acace6ccac46991f P 8feeac9977814b8b901607ec4071efdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:46.307255 581 tablet_replica.cc:330] T f9c42e0ae4dc4e4f96cbb9161babbbce P 8feeac9977814b8b901607ec4071efdf: stopping tablet replica
I20240502 13:59:46.307320 581 raft_consensus.cc:2229] T f9c42e0ae4dc4e4f96cbb9161babbbce P 8feeac9977814b8b901607ec4071efdf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 13:59:46.307391 581 raft_consensus.cc:2258] T f9c42e0ae4dc4e4f96cbb9161babbbce P 8feeac9977814b8b901607ec4071efdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:46.307782 581 tablet_replica.cc:330] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 8feeac9977814b8b901607ec4071efdf: stopping tablet replica
I20240502 13:59:46.307848 581 raft_consensus.cc:2229] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 8feeac9977814b8b901607ec4071efdf [term 1 LEADER]: Raft consensus shutting down.
I20240502 13:59:46.307956 581 raft_consensus.cc:2258] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 8feeac9977814b8b901607ec4071efdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:46.308192 581 tablet_replica.cc:330] T fb8f611a150140b4827890f46869b33f P 8feeac9977814b8b901607ec4071efdf: stopping tablet replica
I20240502 13:59:46.308251 581 raft_consensus.cc:2229] T fb8f611a150140b4827890f46869b33f P 8feeac9977814b8b901607ec4071efdf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 13:59:46.308315 581 raft_consensus.cc:2258] T fb8f611a150140b4827890f46869b33f P 8feeac9977814b8b901607ec4071efdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:46.321341 581 tablet_server.cc:194] TabletServer@127.0.145.66:0 shutdown complete.
I20240502 13:59:46.323511 581 tablet_server.cc:178] TabletServer@127.0.145.67:0 shutting down...
I20240502 13:59:46.327111 581 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 13:59:46.327248 581 tablet_replica.cc:330] T fb8f611a150140b4827890f46869b33f P 0c0b5fc6c9894f7ca9a083e986452dff: stopping tablet replica
I20240502 13:59:46.327348 581 raft_consensus.cc:2229] T fb8f611a150140b4827890f46869b33f P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 13:59:46.327430 581 raft_consensus.cc:2258] T fb8f611a150140b4827890f46869b33f P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:46.327716 581 tablet_replica.cc:330] T 7a8684e4da8f46b6acace6ccac46991f P 0c0b5fc6c9894f7ca9a083e986452dff: stopping tablet replica
I20240502 13:59:46.327790 581 raft_consensus.cc:2229] T 7a8684e4da8f46b6acace6ccac46991f P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 LEADER]: Raft consensus shutting down.
I20240502 13:59:46.327906 581 raft_consensus.cc:2258] T 7a8684e4da8f46b6acace6ccac46991f P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:46.328148 581 tablet_replica.cc:330] T 2a57d4e298464ad580027819241d4735 P 0c0b5fc6c9894f7ca9a083e986452dff: stopping tablet replica
I20240502 13:59:46.328217 581 raft_consensus.cc:2229] T 2a57d4e298464ad580027819241d4735 P 0c0b5fc6c9894f7ca9a083e986452dff [term 2 LEADER]: Raft consensus shutting down.
I20240502 13:59:46.328330 581 raft_consensus.cc:2258] T 2a57d4e298464ad580027819241d4735 P 0c0b5fc6c9894f7ca9a083e986452dff [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:46.328576 581 tablet_replica.cc:330] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 0c0b5fc6c9894f7ca9a083e986452dff: stopping tablet replica
I20240502 13:59:46.328642 581 raft_consensus.cc:2229] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 0c0b5fc6c9894f7ca9a083e986452dff [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 13:59:46.328712 581 raft_consensus.cc:2258] T 2cad5291c5eb4f12a6bbcbcfdb37a0ed P 0c0b5fc6c9894f7ca9a083e986452dff [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:46.329098 581 tablet_replica.cc:330] T f9c42e0ae4dc4e4f96cbb9161babbbce P 0c0b5fc6c9894f7ca9a083e986452dff: stopping tablet replica
I20240502 13:59:46.329166 581 raft_consensus.cc:2229] T f9c42e0ae4dc4e4f96cbb9161babbbce P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 13:59:46.329237 581 raft_consensus.cc:2258] T f9c42e0ae4dc4e4f96cbb9161babbbce P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:46.329566 581 tablet_replica.cc:330] T 55b1c1f11b1449d8954ef0c95a95b6da P 0c0b5fc6c9894f7ca9a083e986452dff: stopping tablet replica
I20240502 13:59:46.329630 581 raft_consensus.cc:2229] T 55b1c1f11b1449d8954ef0c95a95b6da P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 13:59:46.329699 581 raft_consensus.cc:2258] T 55b1c1f11b1449d8954ef0c95a95b6da P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:46.330088 581 tablet_replica.cc:330] T 3b5648798faa4b2997198f954af3d4e8 P 0c0b5fc6c9894f7ca9a083e986452dff: stopping tablet replica
I20240502 13:59:46.330155 581 raft_consensus.cc:2229] T 3b5648798faa4b2997198f954af3d4e8 P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 13:59:46.330226 581 raft_consensus.cc:2258] T 3b5648798faa4b2997198f954af3d4e8 P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:46.330610 581 tablet_replica.cc:330] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 0c0b5fc6c9894f7ca9a083e986452dff: stopping tablet replica
I20240502 13:59:46.330673 581 raft_consensus.cc:2229] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 13:59:46.330739 581 raft_consensus.cc:2258] T 9f1be05bf52a4de19cc2691aa7f73aa6 P 0c0b5fc6c9894f7ca9a083e986452dff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:46.343343 581 tablet_server.cc:194] TabletServer@127.0.145.67:0 shutdown complete.
I20240502 13:59:46.345371 581 tablet_server.cc:178] TabletServer@127.0.145.68:0 shutting down...
I20240502 13:59:46.348031 581 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 13:59:46.359607 581 tablet_server.cc:194] TabletServer@127.0.145.68:0 shutdown complete.
I20240502 13:59:46.361099 581 master.cc:537] Master@127.0.145.126:41703 shutting down...
I20240502 13:59:46.363745 581 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2c01d5eee794439b85e4f9d59d986b98 [term 1 LEADER]: Raft consensus shutting down.
I20240502 13:59:46.363866 581 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2c01d5eee794439b85e4f9d59d986b98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:46.363912 581 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2c01d5eee794439b85e4f9d59d986b98: stopping tablet replica
I20240502 13:59:46.375892 581 master.cc:552] Master@127.0.145.126:41703 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (7216 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20240502 13:59:46.384819 581 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.145.126:42241,127.0.145.125:42957,127.0.145.124:45829
I20240502 13:59:46.385154 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:46.387178 1656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:46.387280 1653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:46.387692 581 server_base.cc:1034] running on GCE node
W20240502 13:59:46.388000 1654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:46.388149 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 13:59:46.388190 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 13:59:46.388209 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658386388209 us; error 0 us; skew 500 ppm
I20240502 13:59:46.389027 581 webserver.cc:460] Webserver started at http://127.0.145.126:45417/ using document root <none> and password file <none>
I20240502 13:59:46.389148 581 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:46.389194 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:46.389261 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:46.389605 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658376531600-581-0/minicluster-data/master-0-root/instance:
uuid: "f1e91bd999f34a2d91e6747145115b6d"
format_stamp: "Formatted at 2024-05-02 13:59:46 on dist-test-slave-1z5j"
I20240502 13:59:46.390811 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 13:59:46.391566 1661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:46.391791 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:46.391853 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658376531600-581-0/minicluster-data/master-0-root
uuid: "f1e91bd999f34a2d91e6747145115b6d"
format_stamp: "Formatted at 2024-05-02 13:59:46 on dist-test-slave-1z5j"
I20240502 13:59:46.391914 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658376531600-581-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658376531600-581-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658376531600-581-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:46.428263 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:46.428580 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:46.431995 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.126:42241
I20240502 13:59:46.432972 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 13:59:46.436450 1713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.126:42241 every 8 connection(s)
I20240502 13:59:46.437028 1714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 13:59:46.437219 1719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:46.437606 1717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:46.438033 581 server_base.cc:1034] running on GCE node
I20240502 13:59:46.438354 1714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.145.126" port: 42241 } has no permanent_uuid. Determining permanent_uuid...
W20240502 13:59:46.439098 1716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:46.439260 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 13:59:46.439304 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 13:59:46.439323 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658386439323 us; error 0 us; skew 500 ppm
I20240502 13:59:46.440100 581 webserver.cc:460] Webserver started at http://127.0.145.125:38657/ using document root <none> and password file <none>
I20240502 13:59:46.440217 581 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:46.440256 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:46.440317 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:46.440603 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658376531600-581-0/minicluster-data/master-1-root/instance:
uuid: "acff40724e6c4c2883c00d4b3fa9fa18"
format_stamp: "Formatted at 2024-05-02 13:59:46 on dist-test-slave-1z5j"
I20240502 13:59:46.441963 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:46.442617 1714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.145.125" port: 42957 } has no permanent_uuid. Determining permanent_uuid...
I20240502 13:59:46.442870 1726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:46.443096 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:46.443161 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658376531600-581-0/minicluster-data/master-1-root
uuid: "acff40724e6c4c2883c00d4b3fa9fa18"
format_stamp: "Formatted at 2024-05-02 13:59:46 on dist-test-slave-1z5j"
I20240502 13:59:46.443221 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658376531600-581-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658376531600-581-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658376531600-581-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240502 13:59:46.443691 1714 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.145.125:42957: Network error: Client connection negotiation failed: client connection to 127.0.145.125:42957: connect: Connection refused (error 111)
I20240502 13:59:46.463112 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:46.463368 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:46.466815 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.125:42957
I20240502 13:59:46.466830 1780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.125:42957 every 8 connection(s)
I20240502 13:59:46.467375 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 13:59:46.467521 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 13:59:46.468681 1783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:46.468657 1781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.145.126" port: 42241 } has no permanent_uuid. Determining permanent_uuid...
W20240502 13:59:46.468833 1784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:46.469163 1786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:46.469554 581 server_base.cc:1034] running on GCE node
I20240502 13:59:46.469662 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 13:59:46.469696 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 13:59:46.469713 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658386469713 us; error 0 us; skew 500 ppm
I20240502 13:59:46.470460 581 webserver.cc:460] Webserver started at http://127.0.145.124:35633/ using document root <none> and password file <none>
I20240502 13:59:46.470577 581 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:46.470615 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:46.470676 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:46.470954 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658376531600-581-0/minicluster-data/master-2-root/instance:
uuid: "79a3c2428c96441d807b79144561c15d"
format_stamp: "Formatted at 2024-05-02 13:59:46 on dist-test-slave-1z5j"
I20240502 13:59:46.472298 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:46.472570 1781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.145.125" port: 42957 } has no permanent_uuid. Determining permanent_uuid...
I20240502 13:59:46.473259 1793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:46.473484 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:46.473546 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658376531600-581-0/minicluster-data/master-2-root
uuid: "79a3c2428c96441d807b79144561c15d"
format_stamp: "Formatted at 2024-05-02 13:59:46 on dist-test-slave-1z5j"
I20240502 13:59:46.473604 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658376531600-581-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658376531600-581-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658376531600-581-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:46.476172 1781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.145.124" port: 45829 } has no permanent_uuid. Determining permanent_uuid...
W20240502 13:59:46.477186 1781 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.145.124:45829: Network error: Client connection negotiation failed: client connection to 127.0.145.124:45829: connect: Connection refused (error 111)
I20240502 13:59:46.482525 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:46.482759 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:46.486114 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.124:45829
I20240502 13:59:46.486133 1846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.124:45829 every 8 connection(s)
I20240502 13:59:46.486462 581 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 13:59:46.486755 1847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:46.487597 1847 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.145.126" port: 42241 } has no permanent_uuid. Determining permanent_uuid...
I20240502 13:59:46.490761 1847 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.145.125" port: 42957 } has no permanent_uuid. Determining permanent_uuid...
I20240502 13:59:46.493667 1847 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.145.124" port: 45829 } has no permanent_uuid. Determining permanent_uuid...
I20240502 13:59:46.497427 1847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d: Bootstrap starting.
I20240502 13:59:46.498057 1847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:46.498855 1847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d: No bootstrap required, opened a new log
I20240502 13:59:46.499158 1847 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e91bd999f34a2d91e6747145115b6d" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 42241 } } peers { permanent_uuid: "acff40724e6c4c2883c00d4b3fa9fa18" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 42957 } } peers { permanent_uuid: "79a3c2428c96441d807b79144561c15d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 45829 } }
I20240502 13:59:46.499233 1847 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:46.499261 1847 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79a3c2428c96441d807b79144561c15d, State: Initialized, Role: FOLLOWER
I20240502 13:59:46.499361 1847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e91bd999f34a2d91e6747145115b6d" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 42241 } } peers { permanent_uuid: "acff40724e6c4c2883c00d4b3fa9fa18" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 42957 } } peers { permanent_uuid: "79a3c2428c96441d807b79144561c15d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 45829 } }
I20240502 13:59:46.499603 1852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e91bd999f34a2d91e6747145115b6d" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 42241 } } peers { permanent_uuid: "acff40724e6c4c2883c00d4b3fa9fa18" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 42957 } } peers { permanent_uuid: "79a3c2428c96441d807b79144561c15d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 45829 } } }
I20240502 13:59:46.499687 1852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d [sys.catalog]: This master's current role is: FOLLOWER
I20240502 13:59:46.499703 1847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d [sys.catalog]: configured and running, proceeding with master startup.
W20240502 13:59:46.501456 1863 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 13:59:46.501540 1863 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 13:59:46.508796 1714 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.145.125" port: 42957 } attempt: 1
I20240502 13:59:46.511590 1714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.145.124" port: 45829 } has no permanent_uuid. Determining permanent_uuid...
I20240502 13:59:46.515081 1714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d: Bootstrap starting.
I20240502 13:59:46.515779 1714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:46.516618 1714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d: No bootstrap required, opened a new log
I20240502 13:59:46.516976 1714 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e91bd999f34a2d91e6747145115b6d" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 42241 } } peers { permanent_uuid: "acff40724e6c4c2883c00d4b3fa9fa18" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 42957 } } peers { permanent_uuid: "79a3c2428c96441d807b79144561c15d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 45829 } }
I20240502 13:59:46.517051 1714 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:46.517072 1714 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e91bd999f34a2d91e6747145115b6d, State: Initialized, Role: FOLLOWER
I20240502 13:59:46.517180 1714 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e91bd999f34a2d91e6747145115b6d" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 42241 } } peers { permanent_uuid: "acff40724e6c4c2883c00d4b3fa9fa18" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 42957 } } peers { permanent_uuid: "79a3c2428c96441d807b79144561c15d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 45829 } }
I20240502 13:59:46.517441 1865 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e91bd999f34a2d91e6747145115b6d" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 42241 } } peers { permanent_uuid: "acff40724e6c4c2883c00d4b3fa9fa18" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 42957 } } peers { permanent_uuid: "79a3c2428c96441d807b79144561c15d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 45829 } } }
I20240502 13:59:46.517521 1865 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d [sys.catalog]: This master's current role is: FOLLOWER
I20240502 13:59:46.517558 1714 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d [sys.catalog]: configured and running, proceeding with master startup.
I20240502 13:59:46.518853 581 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20240502 13:59:46.519238 1876 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 13:59:46.519291 1876 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 13:59:46.523398 1865 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:46.523469 1865 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e91bd999f34a2d91e6747145115b6d" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 42241 } } peers { permanent_uuid: "acff40724e6c4c2883c00d4b3fa9fa18" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 42957 } } peers { permanent_uuid: "79a3c2428c96441d807b79144561c15d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 45829 } }
I20240502 13:59:46.523758 1865 leader_election.cc:288] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acff40724e6c4c2883c00d4b3fa9fa18 (127.0.145.125:42957), 79a3c2428c96441d807b79144561c15d (127.0.145.124:45829)
I20240502 13:59:46.524045 1821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1e91bd999f34a2d91e6747145115b6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79a3c2428c96441d807b79144561c15d" is_pre_election: true
I20240502 13:59:46.524137 1821 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e91bd999f34a2d91e6747145115b6d in term 0.
I20240502 13:59:46.524367 1665 leader_election.cc:302] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79a3c2428c96441d807b79144561c15d, f1e91bd999f34a2d91e6747145115b6d; no voters:
I20240502 13:59:46.524511 1865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:46.524567 1865 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:46.524602 1865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:46.525264 1865 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e91bd999f34a2d91e6747145115b6d" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 42241 } } peers { permanent_uuid: "acff40724e6c4c2883c00d4b3fa9fa18" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 42957 } } peers { permanent_uuid: "79a3c2428c96441d807b79144561c15d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 45829 } }
I20240502 13:59:46.525542 1865 leader_election.cc:288] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d [CANDIDATE]: Term 1 election: Requested vote from peers acff40724e6c4c2883c00d4b3fa9fa18 (127.0.145.125:42957), 79a3c2428c96441d807b79144561c15d (127.0.145.124:45829)
I20240502 13:59:46.525800 1821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1e91bd999f34a2d91e6747145115b6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79a3c2428c96441d807b79144561c15d"
I20240502 13:59:46.525899 1821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:46.526727 1821 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e91bd999f34a2d91e6747145115b6d in term 1.
I20240502 13:59:46.526998 1665 leader_election.cc:302] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79a3c2428c96441d807b79144561c15d, f1e91bd999f34a2d91e6747145115b6d; no voters:
I20240502 13:59:46.527137 1865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:46.527319 1865 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d [term 1 LEADER]: Becoming Leader. State: Replica: f1e91bd999f34a2d91e6747145115b6d, State: Running, Role: LEADER
I20240502 13:59:46.527451 1865 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e91bd999f34a2d91e6747145115b6d" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 42241 } } peers { permanent_uuid: "acff40724e6c4c2883c00d4b3fa9fa18" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 42957 } } peers { permanent_uuid: "79a3c2428c96441d807b79144561c15d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 45829 } }
I20240502 13:59:46.528062 1877 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1e91bd999f34a2d91e6747145115b6d. Latest consensus state: current_term: 1 leader_uuid: "f1e91bd999f34a2d91e6747145115b6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e91bd999f34a2d91e6747145115b6d" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 42241 } } peers { permanent_uuid: "acff40724e6c4c2883c00d4b3fa9fa18" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 42957 } } peers { permanent_uuid: "79a3c2428c96441d807b79144561c15d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 45829 } } }
I20240502 13:59:46.528162 1877 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d [sys.catalog]: This master's current role is: LEADER
I20240502 13:59:46.528420 1879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 13:59:46.529165 1879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 13:59:46.529325 1781 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.145.124" port: 45829 } attempt: 1
I20240502 13:59:46.531141 1821 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d [term 1 FOLLOWER]: Refusing update from remote peer f1e91bd999f34a2d91e6747145115b6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:46.531474 1877 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79a3c2428c96441d807b79144561c15d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:46.532624 1852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1e91bd999f34a2d91e6747145115b6d. Latest consensus state: current_term: 1 leader_uuid: "f1e91bd999f34a2d91e6747145115b6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e91bd999f34a2d91e6747145115b6d" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 42241 } } peers { permanent_uuid: "acff40724e6c4c2883c00d4b3fa9fa18" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 42957 } } peers { permanent_uuid: "79a3c2428c96441d807b79144561c15d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 45829 } } }
I20240502 13:59:46.532722 1852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d [sys.catalog]: This master's current role is: FOLLOWER
I20240502 13:59:46.533658 1781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18: Bootstrap starting.
I20240502 13:59:46.533656 1877 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f1e91bd999f34a2d91e6747145115b6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e91bd999f34a2d91e6747145115b6d" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 42241 } } peers { permanent_uuid: "acff40724e6c4c2883c00d4b3fa9fa18" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 42957 } } peers { permanent_uuid: "79a3c2428c96441d807b79144561c15d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 45829 } } }
I20240502 13:59:46.533766 1877 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d [sys.catalog]: This master's current role is: LEADER
I20240502 13:59:46.534263 1852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f1e91bd999f34a2d91e6747145115b6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e91bd999f34a2d91e6747145115b6d" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 42241 } } peers { permanent_uuid: "acff40724e6c4c2883c00d4b3fa9fa18" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 42957 } } peers { permanent_uuid: "79a3c2428c96441d807b79144561c15d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 45829 } } }
I20240502 13:59:46.534354 1852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d [sys.catalog]: This master's current role is: FOLLOWER
I20240502 13:59:46.534551 1879 catalog_manager.cc:1347] Generated new cluster ID: 95108958c00c40ea84c0ac7122819fcf
I20240502 13:59:46.534600 1879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 13:59:46.534621 1781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:46.535651 1781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18: No bootstrap required, opened a new log
I20240502 13:59:46.536015 1781 raft_consensus.cc:348] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e91bd999f34a2d91e6747145115b6d" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 42241 } } peers { permanent_uuid: "acff40724e6c4c2883c00d4b3fa9fa18" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 42957 } } peers { permanent_uuid: "79a3c2428c96441d807b79144561c15d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 45829 } }
I20240502 13:59:46.536103 1781 raft_consensus.cc:374] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:46.536126 1781 raft_consensus.cc:729] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acff40724e6c4c2883c00d4b3fa9fa18, State: Initialized, Role: FOLLOWER
I20240502 13:59:46.536221 1781 consensus_queue.cc:260] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e91bd999f34a2d91e6747145115b6d" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 42241 } } peers { permanent_uuid: "acff40724e6c4c2883c00d4b3fa9fa18" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 42957 } } peers { permanent_uuid: "79a3c2428c96441d807b79144561c15d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 45829 } }
I20240502 13:59:46.536496 1886 sys_catalog.cc:455] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e91bd999f34a2d91e6747145115b6d" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 42241 } } peers { permanent_uuid: "acff40724e6c4c2883c00d4b3fa9fa18" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 42957 } } peers { permanent_uuid: "79a3c2428c96441d807b79144561c15d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 45829 } } }
I20240502 13:59:46.536594 1886 sys_catalog.cc:458] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 13:59:46.536613 1781 sys_catalog.cc:564] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 13:59:46.536695 1754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1e91bd999f34a2d91e6747145115b6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acff40724e6c4c2883c00d4b3fa9fa18"
I20240502 13:59:46.536799 1754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:46.537525 1754 raft_consensus.cc:2454] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e91bd999f34a2d91e6747145115b6d in term 1.
I20240502 13:59:46.538223 581 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240502 13:59:46.538304 1753 raft_consensus.cc:1261] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [term 1 FOLLOWER]: Refusing update from remote peer f1e91bd999f34a2d91e6747145115b6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:46.538483 1755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1e91bd999f34a2d91e6747145115b6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acff40724e6c4c2883c00d4b3fa9fa18" is_pre_election: true
I20240502 13:59:46.538570 1755 raft_consensus.cc:2362] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f1e91bd999f34a2d91e6747145115b6d in term 1. Re-sending same reply.
I20240502 13:59:46.538647 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:46.538914 1897 catalog_manager.cc:1558] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20240502 13:59:46.538934 1865 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "acff40724e6c4c2883c00d4b3fa9fa18" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 42957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 13:59:46.539245 1897 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 13:59:46.540391 1898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:46.540463 1886 sys_catalog.cc:455] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1e91bd999f34a2d91e6747145115b6d. Latest consensus state: current_term: 1 leader_uuid: "f1e91bd999f34a2d91e6747145115b6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e91bd999f34a2d91e6747145115b6d" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 42241 } } peers { permanent_uuid: "acff40724e6c4c2883c00d4b3fa9fa18" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 42957 } } peers { permanent_uuid: "79a3c2428c96441d807b79144561c15d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 45829 } } }
I20240502 13:59:46.540547 1886 sys_catalog.cc:458] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 13:59:46.540624 1886 sys_catalog.cc:455] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f1e91bd999f34a2d91e6747145115b6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e91bd999f34a2d91e6747145115b6d" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 42241 } } peers { permanent_uuid: "acff40724e6c4c2883c00d4b3fa9fa18" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 42957 } } peers { permanent_uuid: "79a3c2428c96441d807b79144561c15d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 45829 } } }
I20240502 13:59:46.540685 1886 sys_catalog.cc:458] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [sys.catalog]: This master's current role is: FOLLOWER
W20240502 13:59:46.540841 1899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:46.541180 1901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:46.541556 581 server_base.cc:1034] running on GCE node
I20240502 13:59:46.541666 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 13:59:46.541703 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 13:59:46.541723 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658386541722 us; error 0 us; skew 500 ppm
I20240502 13:59:46.542536 581 webserver.cc:460] Webserver started at http://127.0.145.65:46839/ using document root <none> and password file <none>
I20240502 13:59:46.542666 581 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:46.542711 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:46.542773 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:46.543104 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658376531600-581-0/minicluster-data/ts-0-root/instance:
uuid: "08fe7a643daa432e8980c265bc441cdc"
format_stamp: "Formatted at 2024-05-02 13:59:46 on dist-test-slave-1z5j"
I20240502 13:59:46.544262 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:46.545053 1911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:46.545285 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:46.545341 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658376531600-581-0/minicluster-data/ts-0-root
uuid: "08fe7a643daa432e8980c265bc441cdc"
format_stamp: "Formatted at 2024-05-02 13:59:46 on dist-test-slave-1z5j"
I20240502 13:59:46.545393 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658376531600-581-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658376531600-581-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658376531600-581-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:46.547341 1865 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f1e91bd999f34a2d91e6747145115b6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e91bd999f34a2d91e6747145115b6d" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 42241 } } peers { permanent_uuid: "acff40724e6c4c2883c00d4b3fa9fa18" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 42957 } } peers { permanent_uuid: "79a3c2428c96441d807b79144561c15d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 45829 } } }
I20240502 13:59:46.547447 1865 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d [sys.catalog]: This master's current role is: LEADER
I20240502 13:59:46.549151 1879 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 13:59:46.549829 1879 catalog_manager.cc:1504] Loading token signing keys...
I20240502 13:59:46.556499 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:46.556818 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:46.557057 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:46.557448 581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:46.557483 581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:46.557518 581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:46.557546 581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:46.562201 1879 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d: Generated new TSK 0
I20240502 13:59:46.562332 1879 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 13:59:46.562655 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.65:34493
I20240502 13:59:46.563521 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 13:59:46.564632 1976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.65:34493 every 8 connection(s)
W20240502 13:59:46.565356 1983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:46.565538 1982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:46.566229 1985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:46.567523 581 server_base.cc:1034] running on GCE node
I20240502 13:59:46.567679 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 13:59:46.567716 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 13:59:46.567734 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658386567734 us; error 0 us; skew 500 ppm
I20240502 13:59:46.568568 581 webserver.cc:460] Webserver started at http://127.0.145.66:34433/ using document root <none> and password file <none>
I20240502 13:59:46.568703 581 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:46.568753 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:46.568840 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:46.569224 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658376531600-581-0/minicluster-data/ts-1-root/instance:
uuid: "64bee0687c484cc7b3c18e2c36f500e7"
format_stamp: "Formatted at 2024-05-02 13:59:46 on dist-test-slave-1z5j"
I20240502 13:59:46.569336 1977 heartbeater.cc:347] Connected to a master server at 127.0.145.124:45829
I20240502 13:59:46.569430 1977 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:46.569438 1978 heartbeater.cc:347] Connected to a master server at 127.0.145.126:42241
I20240502 13:59:46.569515 1978 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:46.569617 1977 heartbeater.cc:511] Master 127.0.145.124:45829 requested a full tablet report, sending...
I20240502 13:59:46.569700 1978 heartbeater.cc:511] Master 127.0.145.126:42241 requested a full tablet report, sending...
I20240502 13:59:46.570227 1811 ts_manager.cc:194] Registered new tserver with Master: 08fe7a643daa432e8980c265bc441cdc (127.0.145.65:34493)
I20240502 13:59:46.570421 1678 ts_manager.cc:194] Registered new tserver with Master: 08fe7a643daa432e8980c265bc441cdc (127.0.145.65:34493)
I20240502 13:59:46.570852 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 13:59:46.571300 1678 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59090
I20240502 13:59:46.571671 1979 heartbeater.cc:347] Connected to a master server at 127.0.145.125:42957
I20240502 13:59:46.571746 1979 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:46.571758 1992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:46.571977 1979 heartbeater.cc:511] Master 127.0.145.125:42957 requested a full tablet report, sending...
I20240502 13:59:46.572016 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:46.572084 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658376531600-581-0/minicluster-data/ts-1-root
uuid: "64bee0687c484cc7b3c18e2c36f500e7"
format_stamp: "Formatted at 2024-05-02 13:59:46 on dist-test-slave-1z5j"
I20240502 13:59:46.572139 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658376531600-581-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658376531600-581-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658376531600-581-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:46.572549 1745 ts_manager.cc:194] Registered new tserver with Master: 08fe7a643daa432e8980c265bc441cdc (127.0.145.65:34493)
I20240502 13:59:46.579713 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:46.579986 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:46.580209 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:46.580567 581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:46.580601 581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:46.580629 581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:46.580657 581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:46.584666 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.66:40361
I20240502 13:59:46.584728 2056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.66:40361 every 8 connection(s)
I20240502 13:59:46.585501 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:46.587575 2065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:46.587989 2067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:46.588459 581 server_base.cc:1034] running on GCE node
W20240502 13:59:46.588706 2064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:46.588956 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 13:59:46.589005 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 13:59:46.589031 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658386589031 us; error 0 us; skew 500 ppm
I20240502 13:59:46.589417 2058 heartbeater.cc:347] Connected to a master server at 127.0.145.126:42241
I20240502 13:59:46.589501 2058 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:46.589687 2058 heartbeater.cc:511] Master 127.0.145.126:42241 requested a full tablet report, sending...
I20240502 13:59:46.589931 581 webserver.cc:460] Webserver started at http://127.0.145.67:36225/ using document root <none> and password file <none>
I20240502 13:59:46.590066 581 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:46.590114 2057 heartbeater.cc:347] Connected to a master server at 127.0.145.124:45829
I20240502 13:59:46.590119 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:46.590178 1678 ts_manager.cc:194] Registered new tserver with Master: 64bee0687c484cc7b3c18e2c36f500e7 (127.0.145.66:40361)
I20240502 13:59:46.590212 2057 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:46.590278 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:46.590417 2057 heartbeater.cc:511] Master 127.0.145.124:45829 requested a full tablet report, sending...
I20240502 13:59:46.590634 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658376531600-581-0/minicluster-data/ts-2-root/instance:
uuid: "19eb481e8cf344d88c84374a4f837b72"
format_stamp: "Formatted at 2024-05-02 13:59:46 on dist-test-slave-1z5j"
I20240502 13:59:46.590795 1678 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59102
I20240502 13:59:46.590968 1811 ts_manager.cc:194] Registered new tserver with Master: 64bee0687c484cc7b3c18e2c36f500e7 (127.0.145.66:40361)
I20240502 13:59:46.591488 2059 heartbeater.cc:347] Connected to a master server at 127.0.145.125:42957
I20240502 13:59:46.591562 2059 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:46.591675 2059 heartbeater.cc:511] Master 127.0.145.125:42957 requested a full tablet report, sending...
I20240502 13:59:46.592191 1745 ts_manager.cc:194] Registered new tserver with Master: 64bee0687c484cc7b3c18e2c36f500e7 (127.0.145.66:40361)
I20240502 13:59:46.592233 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 13:59:46.593086 2072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:46.593274 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:46.593333 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658376531600-581-0/minicluster-data/ts-2-root
uuid: "19eb481e8cf344d88c84374a4f837b72"
format_stamp: "Formatted at 2024-05-02 13:59:46 on dist-test-slave-1z5j"
I20240502 13:59:46.593389 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658376531600-581-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658376531600-581-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658376531600-581-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:46.602006 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:46.602283 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:46.602523 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:46.602881 581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:46.602916 581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:46.602953 581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:46.602981 581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:46.607033 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.67:39303
I20240502 13:59:46.607086 2136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.67:39303 every 8 connection(s)
I20240502 13:59:46.611506 2139 heartbeater.cc:347] Connected to a master server at 127.0.145.125:42957
I20240502 13:59:46.611572 2137 heartbeater.cc:347] Connected to a master server at 127.0.145.124:45829
I20240502 13:59:46.611589 2139 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:46.611622 2137 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:46.611788 2138 heartbeater.cc:347] Connected to a master server at 127.0.145.126:42241
I20240502 13:59:46.611779 2137 heartbeater.cc:511] Master 127.0.145.124:45829 requested a full tablet report, sending...
I20240502 13:59:46.611778 2139 heartbeater.cc:511] Master 127.0.145.125:42957 requested a full tablet report, sending...
I20240502 13:59:46.611857 2138 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:46.612052 2138 heartbeater.cc:511] Master 127.0.145.126:42241 requested a full tablet report, sending...
I20240502 13:59:46.612387 1811 ts_manager.cc:194] Registered new tserver with Master: 19eb481e8cf344d88c84374a4f837b72 (127.0.145.67:39303)
I20240502 13:59:46.612504 1678 ts_manager.cc:194] Registered new tserver with Master: 19eb481e8cf344d88c84374a4f837b72 (127.0.145.67:39303)
I20240502 13:59:46.612699 1745 ts_manager.cc:194] Registered new tserver with Master: 19eb481e8cf344d88c84374a4f837b72 (127.0.145.67:39303)
I20240502 13:59:46.613065 581 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.00557167s
I20240502 13:59:46.613099 1678 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59116
I20240502 13:59:47.502409 1863 catalog_manager.cc:1259] Loaded cluster ID: 95108958c00c40ea84c0ac7122819fcf
I20240502 13:59:47.502494 1863 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d: loading cluster ID for follower catalog manager: success
I20240502 13:59:47.504195 1863 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d: acquiring CA information for follower catalog manager: success
I20240502 13:59:47.504894 1863 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 13:59:47.539883 1897 catalog_manager.cc:1259] Loaded cluster ID: 95108958c00c40ea84c0ac7122819fcf
I20240502 13:59:47.539955 1897 catalog_manager.cc:1552] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18: loading cluster ID for follower catalog manager: success
I20240502 13:59:47.541088 1897 catalog_manager.cc:1574] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18: acquiring CA information for follower catalog manager: success
I20240502 13:59:47.541558 1897 catalog_manager.cc:1602] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 13:59:47.572141 1978 heartbeater.cc:503] Master 127.0.145.126:42241 was elected leader, sending a full tablet report...
I20240502 13:59:47.591581 2058 heartbeater.cc:503] Master 127.0.145.126:42241 was elected leader, sending a full tablet report...
I20240502 13:59:47.613795 2138 heartbeater.cc:503] Master 127.0.145.126:42241 was elected leader, sending a full tablet report...
I20240502 13:59:48.518976 1875 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 13:59:48.673846 581 test_util.cc:268] Using random seed: 69928242
I20240502 13:59:48.682267 1678 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59130:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 13:59:48.694478 1940 tablet_service.cc:1467] Processing CreateTablet for tablet d5429c5dc2e8483c9f8368dc50292f77 (DEFAULT_TABLE table=test-workload [id=e0474a45120e432bb1b0d403ee2ed8d7]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240502 13:59:48.694761 1940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5429c5dc2e8483c9f8368dc50292f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:48.695546 2100 tablet_service.cc:1467] Processing CreateTablet for tablet 5b46231336fa46a4859a41303ceafbda (DEFAULT_TABLE table=test-workload [id=e0474a45120e432bb1b0d403ee2ed8d7]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240502 13:59:48.695780 2100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b46231336fa46a4859a41303ceafbda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:48.696625 2172 tablet_bootstrap.cc:492] T d5429c5dc2e8483c9f8368dc50292f77 P 08fe7a643daa432e8980c265bc441cdc: Bootstrap starting.
I20240502 13:59:48.696745 2021 tablet_service.cc:1467] Processing CreateTablet for tablet b454546b2a4e41648f9cd111f636f626 (DEFAULT_TABLE table=test-workload [id=e0474a45120e432bb1b0d403ee2ed8d7]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240502 13:59:48.697011 2021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b454546b2a4e41648f9cd111f636f626. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:48.697638 2172 tablet_bootstrap.cc:654] T d5429c5dc2e8483c9f8368dc50292f77 P 08fe7a643daa432e8980c265bc441cdc: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:48.697646 2173 tablet_bootstrap.cc:492] T 5b46231336fa46a4859a41303ceafbda P 19eb481e8cf344d88c84374a4f837b72: Bootstrap starting.
I20240502 13:59:48.698494 2172 tablet_bootstrap.cc:492] T d5429c5dc2e8483c9f8368dc50292f77 P 08fe7a643daa432e8980c265bc441cdc: No bootstrap required, opened a new log
I20240502 13:59:48.698554 2173 tablet_bootstrap.cc:654] T 5b46231336fa46a4859a41303ceafbda P 19eb481e8cf344d88c84374a4f837b72: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:48.698571 2172 ts_tablet_manager.cc:1367] T d5429c5dc2e8483c9f8368dc50292f77 P 08fe7a643daa432e8980c265bc441cdc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:48.699054 2172 raft_consensus.cc:348] T d5429c5dc2e8483c9f8368dc50292f77 P 08fe7a643daa432e8980c265bc441cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fe7a643daa432e8980c265bc441cdc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34493 } }
I20240502 13:59:48.699148 2172 raft_consensus.cc:374] T d5429c5dc2e8483c9f8368dc50292f77 P 08fe7a643daa432e8980c265bc441cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:48.699187 2172 raft_consensus.cc:729] T d5429c5dc2e8483c9f8368dc50292f77 P 08fe7a643daa432e8980c265bc441cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08fe7a643daa432e8980c265bc441cdc, State: Initialized, Role: FOLLOWER
I20240502 13:59:48.699317 2172 consensus_queue.cc:260] T d5429c5dc2e8483c9f8368dc50292f77 P 08fe7a643daa432e8980c265bc441cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fe7a643daa432e8980c265bc441cdc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34493 } }
I20240502 13:59:48.699391 2172 raft_consensus.cc:388] T d5429c5dc2e8483c9f8368dc50292f77 P 08fe7a643daa432e8980c265bc441cdc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:48.699433 2172 raft_consensus.cc:482] T d5429c5dc2e8483c9f8368dc50292f77 P 08fe7a643daa432e8980c265bc441cdc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:48.699479 2172 raft_consensus.cc:3037] T d5429c5dc2e8483c9f8368dc50292f77 P 08fe7a643daa432e8980c265bc441cdc [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:48.699779 2173 tablet_bootstrap.cc:492] T 5b46231336fa46a4859a41303ceafbda P 19eb481e8cf344d88c84374a4f837b72: No bootstrap required, opened a new log
I20240502 13:59:48.699874 2173 ts_tablet_manager.cc:1367] T 5b46231336fa46a4859a41303ceafbda P 19eb481e8cf344d88c84374a4f837b72: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:48.700249 2173 raft_consensus.cc:348] T 5b46231336fa46a4859a41303ceafbda P 19eb481e8cf344d88c84374a4f837b72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19eb481e8cf344d88c84374a4f837b72" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 39303 } }
I20240502 13:59:48.700296 2175 tablet_bootstrap.cc:492] T b454546b2a4e41648f9cd111f636f626 P 64bee0687c484cc7b3c18e2c36f500e7: Bootstrap starting.
I20240502 13:59:48.700348 2173 raft_consensus.cc:374] T 5b46231336fa46a4859a41303ceafbda P 19eb481e8cf344d88c84374a4f837b72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:48.700385 2173 raft_consensus.cc:729] T 5b46231336fa46a4859a41303ceafbda P 19eb481e8cf344d88c84374a4f837b72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19eb481e8cf344d88c84374a4f837b72, State: Initialized, Role: FOLLOWER
I20240502 13:59:48.700547 2173 consensus_queue.cc:260] T 5b46231336fa46a4859a41303ceafbda P 19eb481e8cf344d88c84374a4f837b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19eb481e8cf344d88c84374a4f837b72" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 39303 } }
I20240502 13:59:48.700636 2173 raft_consensus.cc:388] T 5b46231336fa46a4859a41303ceafbda P 19eb481e8cf344d88c84374a4f837b72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:48.700677 2173 raft_consensus.cc:482] T 5b46231336fa46a4859a41303ceafbda P 19eb481e8cf344d88c84374a4f837b72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:48.700723 2173 raft_consensus.cc:3037] T 5b46231336fa46a4859a41303ceafbda P 19eb481e8cf344d88c84374a4f837b72 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:48.701186 2175 tablet_bootstrap.cc:654] T b454546b2a4e41648f9cd111f636f626 P 64bee0687c484cc7b3c18e2c36f500e7: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:48.702039 2175 tablet_bootstrap.cc:492] T b454546b2a4e41648f9cd111f636f626 P 64bee0687c484cc7b3c18e2c36f500e7: No bootstrap required, opened a new log
I20240502 13:59:48.702116 2175 ts_tablet_manager.cc:1367] T b454546b2a4e41648f9cd111f636f626 P 64bee0687c484cc7b3c18e2c36f500e7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:48.702463 2175 raft_consensus.cc:348] T b454546b2a4e41648f9cd111f636f626 P 64bee0687c484cc7b3c18e2c36f500e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64bee0687c484cc7b3c18e2c36f500e7" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 40361 } }
I20240502 13:59:48.702533 2175 raft_consensus.cc:374] T b454546b2a4e41648f9cd111f636f626 P 64bee0687c484cc7b3c18e2c36f500e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:48.702558 2175 raft_consensus.cc:729] T b454546b2a4e41648f9cd111f636f626 P 64bee0687c484cc7b3c18e2c36f500e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64bee0687c484cc7b3c18e2c36f500e7, State: Initialized, Role: FOLLOWER
I20240502 13:59:48.702643 2175 consensus_queue.cc:260] T b454546b2a4e41648f9cd111f636f626 P 64bee0687c484cc7b3c18e2c36f500e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64bee0687c484cc7b3c18e2c36f500e7" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 40361 } }
I20240502 13:59:48.702708 2175 raft_consensus.cc:388] T b454546b2a4e41648f9cd111f636f626 P 64bee0687c484cc7b3c18e2c36f500e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:48.702735 2175 raft_consensus.cc:482] T b454546b2a4e41648f9cd111f636f626 P 64bee0687c484cc7b3c18e2c36f500e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:48.702765 2175 raft_consensus.cc:3037] T b454546b2a4e41648f9cd111f636f626 P 64bee0687c484cc7b3c18e2c36f500e7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:48.732491 2175 raft_consensus.cc:504] T b454546b2a4e41648f9cd111f636f626 P 64bee0687c484cc7b3c18e2c36f500e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64bee0687c484cc7b3c18e2c36f500e7" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 40361 } }
I20240502 13:59:48.732502 2173 raft_consensus.cc:504] T 5b46231336fa46a4859a41303ceafbda P 19eb481e8cf344d88c84374a4f837b72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19eb481e8cf344d88c84374a4f837b72" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 39303 } }
I20240502 13:59:48.732502 2172 raft_consensus.cc:504] T d5429c5dc2e8483c9f8368dc50292f77 P 08fe7a643daa432e8980c265bc441cdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fe7a643daa432e8980c265bc441cdc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34493 } }
I20240502 13:59:48.732693 2172 leader_election.cc:302] T d5429c5dc2e8483c9f8368dc50292f77 P 08fe7a643daa432e8980c265bc441cdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08fe7a643daa432e8980c265bc441cdc; no voters:
I20240502 13:59:48.732693 2173 leader_election.cc:302] T 5b46231336fa46a4859a41303ceafbda P 19eb481e8cf344d88c84374a4f837b72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19eb481e8cf344d88c84374a4f837b72; no voters:
I20240502 13:59:48.732933 2172 leader_election.cc:288] T d5429c5dc2e8483c9f8368dc50292f77 P 08fe7a643daa432e8980c265bc441cdc [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:48.732959 2173 leader_election.cc:288] T 5b46231336fa46a4859a41303ceafbda P 19eb481e8cf344d88c84374a4f837b72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:48.733157 2178 raft_consensus.cc:2781] T d5429c5dc2e8483c9f8368dc50292f77 P 08fe7a643daa432e8980c265bc441cdc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:48.732692 2175 leader_election.cc:302] T b454546b2a4e41648f9cd111f636f626 P 64bee0687c484cc7b3c18e2c36f500e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64bee0687c484cc7b3c18e2c36f500e7; no voters:
I20240502 13:59:48.733356 2172 ts_tablet_manager.cc:1398] T d5429c5dc2e8483c9f8368dc50292f77 P 08fe7a643daa432e8980c265bc441cdc: Time spent starting tablet: real 0.035s user 0.002s sys 0.000s
I20240502 13:59:48.733397 2175 leader_election.cc:288] T b454546b2a4e41648f9cd111f636f626 P 64bee0687c484cc7b3c18e2c36f500e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:48.733498 2178 raft_consensus.cc:686] T d5429c5dc2e8483c9f8368dc50292f77 P 08fe7a643daa432e8980c265bc441cdc [term 1 LEADER]: Becoming Leader. State: Replica: 08fe7a643daa432e8980c265bc441cdc, State: Running, Role: LEADER
I20240502 13:59:48.733608 2175 ts_tablet_manager.cc:1398] T b454546b2a4e41648f9cd111f636f626 P 64bee0687c484cc7b3c18e2c36f500e7: Time spent starting tablet: real 0.031s user 0.002s sys 0.000s
I20240502 13:59:48.733695 2178 consensus_queue.cc:237] T d5429c5dc2e8483c9f8368dc50292f77 P 08fe7a643daa432e8980c265bc441cdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fe7a643daa432e8980c265bc441cdc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34493 } }
I20240502 13:59:48.734040 2179 raft_consensus.cc:2781] T 5b46231336fa46a4859a41303ceafbda P 19eb481e8cf344d88c84374a4f837b72 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:48.733207 2173 ts_tablet_manager.cc:1398] T 5b46231336fa46a4859a41303ceafbda P 19eb481e8cf344d88c84374a4f837b72: Time spent starting tablet: real 0.033s user 0.002s sys 0.001s
I20240502 13:59:48.734306 2184 raft_consensus.cc:2781] T b454546b2a4e41648f9cd111f636f626 P 64bee0687c484cc7b3c18e2c36f500e7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:48.735101 2184 raft_consensus.cc:686] T b454546b2a4e41648f9cd111f636f626 P 64bee0687c484cc7b3c18e2c36f500e7 [term 1 LEADER]: Becoming Leader. State: Replica: 64bee0687c484cc7b3c18e2c36f500e7, State: Running, Role: LEADER
I20240502 13:59:48.735267 2184 consensus_queue.cc:237] T b454546b2a4e41648f9cd111f636f626 P 64bee0687c484cc7b3c18e2c36f500e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64bee0687c484cc7b3c18e2c36f500e7" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 40361 } }
I20240502 13:59:48.735705 2179 raft_consensus.cc:686] T 5b46231336fa46a4859a41303ceafbda P 19eb481e8cf344d88c84374a4f837b72 [term 1 LEADER]: Becoming Leader. State: Replica: 19eb481e8cf344d88c84374a4f837b72, State: Running, Role: LEADER
I20240502 13:59:48.735862 2179 consensus_queue.cc:237] T 5b46231336fa46a4859a41303ceafbda P 19eb481e8cf344d88c84374a4f837b72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19eb481e8cf344d88c84374a4f837b72" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 39303 } }
I20240502 13:59:48.736538 1678 catalog_manager.cc:5484] T d5429c5dc2e8483c9f8368dc50292f77 P 08fe7a643daa432e8980c265bc441cdc reported cstate change: term changed from 0 to 1, leader changed from <none> to 08fe7a643daa432e8980c265bc441cdc (127.0.145.65). New cstate: current_term: 1 leader_uuid: "08fe7a643daa432e8980c265bc441cdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fe7a643daa432e8980c265bc441cdc" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34493 } health_report { overall_health: HEALTHY } } }
I20240502 13:59:48.736910 1677 catalog_manager.cc:5484] T b454546b2a4e41648f9cd111f636f626 P 64bee0687c484cc7b3c18e2c36f500e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64bee0687c484cc7b3c18e2c36f500e7 (127.0.145.66). New cstate: current_term: 1 leader_uuid: "64bee0687c484cc7b3c18e2c36f500e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64bee0687c484cc7b3c18e2c36f500e7" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 40361 } health_report { overall_health: HEALTHY } } }
I20240502 13:59:48.738672 1676 catalog_manager.cc:5484] T 5b46231336fa46a4859a41303ceafbda P 19eb481e8cf344d88c84374a4f837b72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19eb481e8cf344d88c84374a4f837b72 (127.0.145.67). New cstate: current_term: 1 leader_uuid: "19eb481e8cf344d88c84374a4f837b72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19eb481e8cf344d88c84374a4f837b72" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 39303 } health_report { overall_health: HEALTHY } } }
I20240502 13:59:48.753861 581 master.cc:537] Master@127.0.145.126:42241 shutting down...
I20240502 13:59:48.756577 581 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d [term 1 LEADER]: Raft consensus shutting down.
I20240502 13:59:48.756726 581 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:48.756803 581 tablet_replica.cc:330] T 00000000000000000000000000000000 P f1e91bd999f34a2d91e6747145115b6d: stopping tablet replica
I20240502 13:59:48.769558 581 master.cc:552] Master@127.0.145.126:42241 shutdown complete.
W20240502 13:59:49.745256 2058 heartbeater.cc:644] Failed to heartbeat to 127.0.145.126:42241 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.145.126:42241: connect: Connection refused (error 111)
I20240502 13:59:50.244462 2199 raft_consensus.cc:482] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f1e91bd999f34a2d91e6747145115b6d)
I20240502 13:59:50.244606 2199 raft_consensus.cc:504] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e91bd999f34a2d91e6747145115b6d" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 42241 } } peers { permanent_uuid: "acff40724e6c4c2883c00d4b3fa9fa18" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 42957 } } peers { permanent_uuid: "79a3c2428c96441d807b79144561c15d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 45829 } }
I20240502 13:59:50.245074 2200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f1e91bd999f34a2d91e6747145115b6d)
I20240502 13:59:50.245069 2199 leader_election.cc:288] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1e91bd999f34a2d91e6747145115b6d (127.0.145.126:42241), 79a3c2428c96441d807b79144561c15d (127.0.145.124:45829)
I20240502 13:59:50.245285 2200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e91bd999f34a2d91e6747145115b6d" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 42241 } } peers { permanent_uuid: "acff40724e6c4c2883c00d4b3fa9fa18" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 42957 } } peers { permanent_uuid: "79a3c2428c96441d807b79144561c15d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 45829 } }
I20240502 13:59:50.245496 1821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "acff40724e6c4c2883c00d4b3fa9fa18" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "79a3c2428c96441d807b79144561c15d" is_pre_election: true
I20240502 13:59:50.245615 1821 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acff40724e6c4c2883c00d4b3fa9fa18 in term 1.
I20240502 13:59:50.245985 1731 leader_election.cc:302] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79a3c2428c96441d807b79144561c15d, acff40724e6c4c2883c00d4b3fa9fa18; no voters:
I20240502 13:59:50.246150 2199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 13:59:50.246192 2200 leader_election.cc:288] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1e91bd999f34a2d91e6747145115b6d (127.0.145.126:42241), acff40724e6c4c2883c00d4b3fa9fa18 (127.0.145.125:42957)
I20240502 13:59:50.246348 1755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79a3c2428c96441d807b79144561c15d" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "acff40724e6c4c2883c00d4b3fa9fa18" is_pre_election: true
I20240502 13:59:50.246203 2199 raft_consensus.cc:482] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f1e91bd999f34a2d91e6747145115b6d)
I20240502 13:59:50.246476 2199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [term 1 FOLLOWER]: Advancing to term 2
W20240502 13:59:50.246670 1730 leader_election.cc:334] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1e91bd999f34a2d91e6747145115b6d (127.0.145.126:42241): Network error: Client connection negotiation failed: client connection to 127.0.145.126:42241: connect: Connection refused (error 111)
W20240502 13:59:50.246946 1796 leader_election.cc:334] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1e91bd999f34a2d91e6747145115b6d (127.0.145.126:42241): Network error: Client connection negotiation failed: client connection to 127.0.145.126:42241: connect: Connection refused (error 111)
I20240502 13:59:50.271510 2199 raft_consensus.cc:504] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e91bd999f34a2d91e6747145115b6d" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 42241 } } peers { permanent_uuid: "acff40724e6c4c2883c00d4b3fa9fa18" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 42957 } } peers { permanent_uuid: "79a3c2428c96441d807b79144561c15d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 45829 } }
I20240502 13:59:50.271826 1755 raft_consensus.cc:2379] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 79a3c2428c96441d807b79144561c15d in current term 2: Already voted for candidate acff40724e6c4c2883c00d4b3fa9fa18 in this term.
I20240502 13:59:50.272004 2199 leader_election.cc:288] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [CANDIDATE]: Term 2 election: Requested vote from peers f1e91bd999f34a2d91e6747145115b6d (127.0.145.126:42241), 79a3c2428c96441d807b79144561c15d (127.0.145.124:45829)
I20240502 13:59:50.272373 1794 leader_election.cc:302] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 79a3c2428c96441d807b79144561c15d; no voters: acff40724e6c4c2883c00d4b3fa9fa18, f1e91bd999f34a2d91e6747145115b6d
I20240502 13:59:50.272444 1821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "acff40724e6c4c2883c00d4b3fa9fa18" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "79a3c2428c96441d807b79144561c15d"
I20240502 13:59:50.272539 1821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d [term 1 FOLLOWER]: Advancing to term 2
W20240502 13:59:50.273083 1730 leader_election.cc:334] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f1e91bd999f34a2d91e6747145115b6d (127.0.145.126:42241): Network error: Client connection negotiation failed: client connection to 127.0.145.126:42241: connect: Connection refused (error 111)
I20240502 13:59:50.273540 1821 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acff40724e6c4c2883c00d4b3fa9fa18 in term 2.
I20240502 13:59:50.273659 2200 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 13:59:50.273860 1731 leader_election.cc:302] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 79a3c2428c96441d807b79144561c15d, acff40724e6c4c2883c00d4b3fa9fa18; no voters: f1e91bd999f34a2d91e6747145115b6d
I20240502 13:59:50.274032 2199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 13:59:50.274242 2199 raft_consensus.cc:686] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [term 2 LEADER]: Becoming Leader. State: Replica: acff40724e6c4c2883c00d4b3fa9fa18, State: Running, Role: LEADER
I20240502 13:59:50.274376 2199 consensus_queue.cc:237] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e91bd999f34a2d91e6747145115b6d" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 42241 } } peers { permanent_uuid: "acff40724e6c4c2883c00d4b3fa9fa18" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 42957 } } peers { permanent_uuid: "79a3c2428c96441d807b79144561c15d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 45829 } }
I20240502 13:59:50.275060 2205 sys_catalog.cc:455] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acff40724e6c4c2883c00d4b3fa9fa18. Latest consensus state: current_term: 2 leader_uuid: "acff40724e6c4c2883c00d4b3fa9fa18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e91bd999f34a2d91e6747145115b6d" member_type: VOTER last_known_addr { host: "127.0.145.126" port: 42241 } } peers { permanent_uuid: "acff40724e6c4c2883c00d4b3fa9fa18" member_type: VOTER last_known_addr { host: "127.0.145.125" port: 42957 } } peers { permanent_uuid: "79a3c2428c96441d807b79144561c15d" member_type: VOTER last_known_addr { host: "127.0.145.124" port: 45829 } } }
I20240502 13:59:50.275185 2205 sys_catalog.cc:458] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [sys.catalog]: This master's current role is: LEADER
I20240502 13:59:50.275452 2207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 13:59:50.276504 2207 catalog_manager.cc:669] Loaded metadata for table test-workload [id=e0474a45120e432bb1b0d403ee2ed8d7]
I20240502 13:59:50.277251 2207 tablet_loader.cc:96] loaded metadata for tablet 5b46231336fa46a4859a41303ceafbda (table test-workload [id=e0474a45120e432bb1b0d403ee2ed8d7])
I20240502 13:59:50.277434 2207 tablet_loader.cc:96] loaded metadata for tablet b454546b2a4e41648f9cd111f636f626 (table test-workload [id=e0474a45120e432bb1b0d403ee2ed8d7])
I20240502 13:59:50.277590 2207 tablet_loader.cc:96] loaded metadata for tablet d5429c5dc2e8483c9f8368dc50292f77 (table test-workload [id=e0474a45120e432bb1b0d403ee2ed8d7])
I20240502 13:59:50.277688 2207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 13:59:50.278057 2207 catalog_manager.cc:1259] Loaded cluster ID: 95108958c00c40ea84c0ac7122819fcf
I20240502 13:59:50.278102 2207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 13:59:50.278599 2207 catalog_manager.cc:1504] Loading token signing keys...
I20240502 13:59:50.278976 2207 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18: Loaded TSK: 0
I20240502 13:59:50.279393 2207 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 13:59:50.538586 1896 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 13:59:50.538684 1896 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 13:59:50.633203 581 tablet_server.cc:178] TabletServer@127.0.145.65:0 shutting down...
I20240502 13:59:50.636689 581 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 13:59:50.636902 581 tablet_replica.cc:330] T d5429c5dc2e8483c9f8368dc50292f77 P 08fe7a643daa432e8980c265bc441cdc: stopping tablet replica
I20240502 13:59:50.637019 581 raft_consensus.cc:2229] T d5429c5dc2e8483c9f8368dc50292f77 P 08fe7a643daa432e8980c265bc441cdc [term 1 LEADER]: Raft consensus shutting down.
I20240502 13:59:50.637144 581 raft_consensus.cc:2258] T d5429c5dc2e8483c9f8368dc50292f77 P 08fe7a643daa432e8980c265bc441cdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:50.650326 581 tablet_server.cc:194] TabletServer@127.0.145.65:0 shutdown complete.
I20240502 13:59:50.652354 581 tablet_server.cc:178] TabletServer@127.0.145.66:0 shutting down...
W20240502 13:59:50.656023 1730 consensus_peers.cc:485] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 -> Peer f1e91bd999f34a2d91e6747145115b6d (127.0.145.126:42241): Couldn't send request to peer f1e91bd999f34a2d91e6747145115b6d. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.126:42241: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 13:59:50.656489 581 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 13:59:50.656643 581 tablet_replica.cc:330] T b454546b2a4e41648f9cd111f636f626 P 64bee0687c484cc7b3c18e2c36f500e7: stopping tablet replica
I20240502 13:59:50.656740 581 raft_consensus.cc:2229] T b454546b2a4e41648f9cd111f636f626 P 64bee0687c484cc7b3c18e2c36f500e7 [term 1 LEADER]: Raft consensus shutting down.
I20240502 13:59:50.656877 581 raft_consensus.cc:2258] T b454546b2a4e41648f9cd111f636f626 P 64bee0687c484cc7b3c18e2c36f500e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:50.668954 581 tablet_server.cc:194] TabletServer@127.0.145.66:0 shutdown complete.
I20240502 13:59:50.670784 581 tablet_server.cc:178] TabletServer@127.0.145.67:0 shutting down...
I20240502 13:59:50.675045 581 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 13:59:50.675194 581 tablet_replica.cc:330] T 5b46231336fa46a4859a41303ceafbda P 19eb481e8cf344d88c84374a4f837b72: stopping tablet replica
I20240502 13:59:50.675297 581 raft_consensus.cc:2229] T 5b46231336fa46a4859a41303ceafbda P 19eb481e8cf344d88c84374a4f837b72 [term 1 LEADER]: Raft consensus shutting down.
I20240502 13:59:50.675390 581 raft_consensus.cc:2258] T 5b46231336fa46a4859a41303ceafbda P 19eb481e8cf344d88c84374a4f837b72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:50.687315 581 tablet_server.cc:194] TabletServer@127.0.145.67:0 shutdown complete.
I20240502 13:59:50.688906 581 master.cc:537] Master@127.0.145.125:42957 shutting down...
I20240502 13:59:50.691723 581 raft_consensus.cc:2229] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [term 2 LEADER]: Raft consensus shutting down.
I20240502 13:59:50.691892 581 pending_rounds.cc:62] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18: Trying to abort 1 pending ops.
I20240502 13:59:50.691951 581 pending_rounds.cc:68] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18: Aborting op as it isn't in flight: id { term: 2 index: 10 } timestamp: 7023240766565208064 op_type: NO_OP noop_request { }
I20240502 13:59:50.692035 581 raft_consensus.cc:2866] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 13:59:50.692078 581 raft_consensus.cc:2258] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:50.692155 581 tablet_replica.cc:330] T 00000000000000000000000000000000 P acff40724e6c4c2883c00d4b3fa9fa18: stopping tablet replica
I20240502 13:59:50.704488 581 master.cc:552] Master@127.0.145.125:42957 shutdown complete.
I20240502 13:59:50.707082 581 master.cc:537] Master@127.0.145.124:45829 shutting down...
I20240502 13:59:50.709992 581 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 13:59:50.710120 581 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 13:59:50.710175 581 tablet_replica.cc:330] T 00000000000000000000000000000000 P 79a3c2428c96441d807b79144561c15d: stopping tablet replica
I20240502 13:59:50.722170 581 master.cc:552] Master@127.0.145.124:45829 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (4343 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20240502 13:59:50.728514 581 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.145.126:41589
I20240502 13:59:50.728826 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:50.730628 2208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:50.731104 2211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:50.731509 2209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:50.731927 581 server_base.cc:1034] running on GCE node
I20240502 13:59:50.732065 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 13:59:50.732101 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 13:59:50.732115 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658390732114 us; error 0 us; skew 500 ppm
I20240502 13:59:50.732936 581 webserver.cc:460] Webserver started at http://127.0.145.126:35223/ using document root <none> and password file <none>
I20240502 13:59:50.733063 581 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:50.733103 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:50.733170 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:50.733526 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658376531600-581-0/minicluster-data/master-0-root/instance:
uuid: "468d6d29941242bdab578cd9b29ccf2f"
format_stamp: "Formatted at 2024-05-02 13:59:50 on dist-test-slave-1z5j"
I20240502 13:59:50.734788 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 13:59:50.735523 2216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:50.735759 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:50.735821 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658376531600-581-0/minicluster-data/master-0-root
uuid: "468d6d29941242bdab578cd9b29ccf2f"
format_stamp: "Formatted at 2024-05-02 13:59:50 on dist-test-slave-1z5j"
I20240502 13:59:50.735880 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658376531600-581-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658376531600-581-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658376531600-581-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:50.758383 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:50.758668 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:50.762197 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.126:41589
I20240502 13:59:50.764660 2269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.126:41589 every 8 connection(s)
I20240502 13:59:50.765089 2270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:50.766690 2270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 468d6d29941242bdab578cd9b29ccf2f: Bootstrap starting.
I20240502 13:59:50.767380 2270 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 468d6d29941242bdab578cd9b29ccf2f: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:50.768478 2270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 468d6d29941242bdab578cd9b29ccf2f: No bootstrap required, opened a new log
I20240502 13:59:50.768894 2270 raft_consensus.cc:348] T 00000000000000000000000000000000 P 468d6d29941242bdab578cd9b29ccf2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "468d6d29941242bdab578cd9b29ccf2f" member_type: VOTER }
I20240502 13:59:50.768991 2270 raft_consensus.cc:374] T 00000000000000000000000000000000 P 468d6d29941242bdab578cd9b29ccf2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:50.769026 2270 raft_consensus.cc:729] T 00000000000000000000000000000000 P 468d6d29941242bdab578cd9b29ccf2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 468d6d29941242bdab578cd9b29ccf2f, State: Initialized, Role: FOLLOWER
I20240502 13:59:50.769145 2270 consensus_queue.cc:260] T 00000000000000000000000000000000 P 468d6d29941242bdab578cd9b29ccf2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "468d6d29941242bdab578cd9b29ccf2f" member_type: VOTER }
I20240502 13:59:50.769212 2270 raft_consensus.cc:388] T 00000000000000000000000000000000 P 468d6d29941242bdab578cd9b29ccf2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:50.769245 2270 raft_consensus.cc:482] T 00000000000000000000000000000000 P 468d6d29941242bdab578cd9b29ccf2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:50.769289 2270 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 468d6d29941242bdab578cd9b29ccf2f [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:50.770025 2270 raft_consensus.cc:504] T 00000000000000000000000000000000 P 468d6d29941242bdab578cd9b29ccf2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "468d6d29941242bdab578cd9b29ccf2f" member_type: VOTER }
I20240502 13:59:50.770151 2270 leader_election.cc:302] T 00000000000000000000000000000000 P 468d6d29941242bdab578cd9b29ccf2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 468d6d29941242bdab578cd9b29ccf2f; no voters:
I20240502 13:59:50.770315 2270 leader_election.cc:288] T 00000000000000000000000000000000 P 468d6d29941242bdab578cd9b29ccf2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:50.770399 2273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 468d6d29941242bdab578cd9b29ccf2f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:50.770571 2273 raft_consensus.cc:686] T 00000000000000000000000000000000 P 468d6d29941242bdab578cd9b29ccf2f [term 1 LEADER]: Becoming Leader. State: Replica: 468d6d29941242bdab578cd9b29ccf2f, State: Running, Role: LEADER
I20240502 13:59:50.770709 2270 sys_catalog.cc:564] T 00000000000000000000000000000000 P 468d6d29941242bdab578cd9b29ccf2f [sys.catalog]: configured and running, proceeding with master startup.
I20240502 13:59:50.770737 2273 consensus_queue.cc:237] T 00000000000000000000000000000000 P 468d6d29941242bdab578cd9b29ccf2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "468d6d29941242bdab578cd9b29ccf2f" member_type: VOTER }
I20240502 13:59:50.771140 2275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 468d6d29941242bdab578cd9b29ccf2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "468d6d29941242bdab578cd9b29ccf2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "468d6d29941242bdab578cd9b29ccf2f" member_type: VOTER } }
I20240502 13:59:50.771248 2275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 468d6d29941242bdab578cd9b29ccf2f [sys.catalog]: This master's current role is: LEADER
I20240502 13:59:50.771140 2273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 468d6d29941242bdab578cd9b29ccf2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 468d6d29941242bdab578cd9b29ccf2f. Latest consensus state: current_term: 1 leader_uuid: "468d6d29941242bdab578cd9b29ccf2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "468d6d29941242bdab578cd9b29ccf2f" member_type: VOTER } }
I20240502 13:59:50.771296 2273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 468d6d29941242bdab578cd9b29ccf2f [sys.catalog]: This master's current role is: LEADER
I20240502 13:59:50.771606 2278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 13:59:50.772518 2278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 13:59:50.772714 581 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 13:59:50.774286 2278 catalog_manager.cc:1347] Generated new cluster ID: a60fbfb2fd6049608950fa40a98032c7
I20240502 13:59:50.774356 2278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 13:59:50.790628 2278 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 13:59:50.791136 2278 catalog_manager.cc:1504] Loading token signing keys...
I20240502 13:59:50.796350 2278 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 468d6d29941242bdab578cd9b29ccf2f: Generated new TSK 0
I20240502 13:59:50.796489 2278 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 13:59:50.804817 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:50.807140 2292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:50.807595 2294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:50.807904 2291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:50.808300 581 server_base.cc:1034] running on GCE node
I20240502 13:59:50.808424 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 13:59:50.808455 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 13:59:50.808468 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658390808468 us; error 0 us; skew 500 ppm
I20240502 13:59:50.809302 581 webserver.cc:460] Webserver started at http://127.0.145.65:37223/ using document root <none> and password file <none>
I20240502 13:59:50.809422 581 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:50.809459 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:50.809512 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:50.809819 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658376531600-581-0/minicluster-data/ts-0-root/instance:
uuid: "661df3a5c18845898204ed4eab51c970"
format_stamp: "Formatted at 2024-05-02 13:59:50 on dist-test-slave-1z5j"
I20240502 13:59:50.811072 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:50.811853 2299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:50.812413 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:50.812482 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658376531600-581-0/minicluster-data/ts-0-root
uuid: "661df3a5c18845898204ed4eab51c970"
format_stamp: "Formatted at 2024-05-02 13:59:50 on dist-test-slave-1z5j"
I20240502 13:59:50.812542 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658376531600-581-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658376531600-581-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658376531600-581-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:50.830497 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:50.830852 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:50.831107 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:50.831545 581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:50.831583 581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:50.831621 581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:50.831645 581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:50.835723 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.65:41099
I20240502 13:59:50.835758 2362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.65:41099 every 8 connection(s)
I20240502 13:59:50.836544 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:50.838587 2367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:50.839087 2368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:50.839553 2370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:50.839928 581 server_base.cc:1034] running on GCE node
I20240502 13:59:50.840093 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 13:59:50.840130 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 13:59:50.840149 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658390840148 us; error 0 us; skew 500 ppm
I20240502 13:59:50.840617 2363 heartbeater.cc:347] Connected to a master server at 127.0.145.126:41589
I20240502 13:59:50.840714 2363 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:50.840926 2363 heartbeater.cc:511] Master 127.0.145.126:41589 requested a full tablet report, sending...
I20240502 13:59:50.840978 581 webserver.cc:460] Webserver started at http://127.0.145.66:46491/ using document root <none> and password file <none>
I20240502 13:59:50.841117 581 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:50.841161 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:50.841230 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:50.841499 2233 ts_manager.cc:194] Registered new tserver with Master: 661df3a5c18845898204ed4eab51c970 (127.0.145.65:41099)
I20240502 13:59:50.841590 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658376531600-581-0/minicluster-data/ts-1-root/instance:
uuid: "6a11d8cedf8848539f3de99c6a20ccc5"
format_stamp: "Formatted at 2024-05-02 13:59:50 on dist-test-slave-1z5j"
I20240502 13:59:50.842192 2233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35172
I20240502 13:59:50.842901 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 13:59:50.843679 2375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:50.843891 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:50.843948 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658376531600-581-0/minicluster-data/ts-1-root
uuid: "6a11d8cedf8848539f3de99c6a20ccc5"
format_stamp: "Formatted at 2024-05-02 13:59:50 on dist-test-slave-1z5j"
I20240502 13:59:50.844009 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658376531600-581-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658376531600-581-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658376531600-581-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:50.872066 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:50.872527 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:50.872907 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:50.873479 581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:50.873520 581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:50.873551 581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:50.873579 581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:50.877333 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.66:41707
I20240502 13:59:50.877383 2439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.66:41707 every 8 connection(s)
I20240502 13:59:50.878068 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:50.879894 2443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:50.880432 2444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:50.880923 2446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:50.881327 581 server_base.cc:1034] running on GCE node
I20240502 13:59:50.881533 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 13:59:50.881577 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 13:59:50.881596 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658390881595 us; error 0 us; skew 500 ppm
I20240502 13:59:50.881860 2440 heartbeater.cc:347] Connected to a master server at 127.0.145.126:41589
I20240502 13:59:50.881956 2440 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:50.882174 2440 heartbeater.cc:511] Master 127.0.145.126:41589 requested a full tablet report, sending...
I20240502 13:59:50.882539 581 webserver.cc:460] Webserver started at http://127.0.145.67:46847/ using document root <none> and password file <none>
I20240502 13:59:50.882688 581 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:50.882742 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:50.882812 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:50.883255 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658376531600-581-0/minicluster-data/ts-2-root/instance:
uuid: "88df0a071a0546aabeb57557af12bcff"
format_stamp: "Formatted at 2024-05-02 13:59:50 on dist-test-slave-1z5j"
I20240502 13:59:50.884953 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240502 13:59:50.885007 2233 ts_manager.cc:194] Registered new tserver with Master: 6a11d8cedf8848539f3de99c6a20ccc5 (127.0.145.66:41707)
I20240502 13:59:50.885675 2233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35186
I20240502 13:59:50.885974 2451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:50.886708 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:50.886791 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658376531600-581-0/minicluster-data/ts-2-root
uuid: "88df0a071a0546aabeb57557af12bcff"
format_stamp: "Formatted at 2024-05-02 13:59:50 on dist-test-slave-1z5j"
I20240502 13:59:50.886857 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658376531600-581-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658376531600-581-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658376531600-581-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:50.911422 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:50.911744 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:50.912015 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:50.912446 581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:50.912496 581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:50.912536 581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:50.912564 581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:50.916657 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.67:43723
I20240502 13:59:50.916707 2515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.67:43723 every 8 connection(s)
I20240502 13:59:50.920408 2516 heartbeater.cc:347] Connected to a master server at 127.0.145.126:41589
I20240502 13:59:50.920492 2516 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:50.920686 2516 heartbeater.cc:511] Master 127.0.145.126:41589 requested a full tablet report, sending...
I20240502 13:59:50.921221 2233 ts_manager.cc:194] Registered new tserver with Master: 88df0a071a0546aabeb57557af12bcff (127.0.145.67:43723)
I20240502 13:59:50.921525 581 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004507153s
I20240502 13:59:50.921802 2233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35192
I20240502 13:59:51.843036 2363 heartbeater.cc:503] Master 127.0.145.126:41589 was elected leader, sending a full tablet report...
I20240502 13:59:51.886632 2440 heartbeater.cc:503] Master 127.0.145.126:41589 was elected leader, sending a full tablet report...
I20240502 13:59:51.922602 2516 heartbeater.cc:503] Master 127.0.145.126:41589 was elected leader, sending a full tablet report...
I20240502 13:59:51.947049 581 test_util.cc:268] Using random seed: 73201446
I20240502 13:59:51.958313 2233 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35198:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 13:59:51.958791 2233 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 13:59:51.968327 2403 tablet_service.cc:1467] Processing CreateTablet for tablet a0f5fad9037a43c686c6638e2bcb9ecb (DEFAULT_TABLE table=test-workload [id=110b644d079c43f5a2dadd853b562fdd]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 13:59:51.968488 2327 tablet_service.cc:1467] Processing CreateTablet for tablet 4c8e99e6b3a2407296d6fedfe6516947 (DEFAULT_TABLE table=test-workload [id=110b644d079c43f5a2dadd853b562fdd]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 13:59:51.968659 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0f5fad9037a43c686c6638e2bcb9ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:51.968796 2327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c8e99e6b3a2407296d6fedfe6516947. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:51.968315 2326 tablet_service.cc:1467] Processing CreateTablet for tablet a0f5fad9037a43c686c6638e2bcb9ecb (DEFAULT_TABLE table=test-workload [id=110b644d079c43f5a2dadd853b562fdd]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 13:59:51.969125 2326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0f5fad9037a43c686c6638e2bcb9ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:51.969259 2479 tablet_service.cc:1467] Processing CreateTablet for tablet 4c8e99e6b3a2407296d6fedfe6516947 (DEFAULT_TABLE table=test-workload [id=110b644d079c43f5a2dadd853b562fdd]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 13:59:51.969480 2479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c8e99e6b3a2407296d6fedfe6516947. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:51.968315 2404 tablet_service.cc:1467] Processing CreateTablet for tablet 4c8e99e6b3a2407296d6fedfe6516947 (DEFAULT_TABLE table=test-workload [id=110b644d079c43f5a2dadd853b562fdd]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 13:59:51.971220 2537 tablet_bootstrap.cc:492] T 4c8e99e6b3a2407296d6fedfe6516947 P 661df3a5c18845898204ed4eab51c970: Bootstrap starting.
I20240502 13:59:51.971329 2404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c8e99e6b3a2407296d6fedfe6516947. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:51.971498 2478 tablet_service.cc:1467] Processing CreateTablet for tablet a0f5fad9037a43c686c6638e2bcb9ecb (DEFAULT_TABLE table=test-workload [id=110b644d079c43f5a2dadd853b562fdd]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 13:59:51.971725 2478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0f5fad9037a43c686c6638e2bcb9ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:51.972162 2537 tablet_bootstrap.cc:654] T 4c8e99e6b3a2407296d6fedfe6516947 P 661df3a5c18845898204ed4eab51c970: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:51.972851 2538 tablet_bootstrap.cc:492] T 4c8e99e6b3a2407296d6fedfe6516947 P 88df0a071a0546aabeb57557af12bcff: Bootstrap starting.
I20240502 13:59:51.973628 2538 tablet_bootstrap.cc:654] T 4c8e99e6b3a2407296d6fedfe6516947 P 88df0a071a0546aabeb57557af12bcff: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:51.974225 2537 tablet_bootstrap.cc:492] T 4c8e99e6b3a2407296d6fedfe6516947 P 661df3a5c18845898204ed4eab51c970: No bootstrap required, opened a new log
I20240502 13:59:51.974305 2537 ts_tablet_manager.cc:1367] T 4c8e99e6b3a2407296d6fedfe6516947 P 661df3a5c18845898204ed4eab51c970: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 13:59:51.974462 2538 tablet_bootstrap.cc:492] T 4c8e99e6b3a2407296d6fedfe6516947 P 88df0a071a0546aabeb57557af12bcff: No bootstrap required, opened a new log
I20240502 13:59:51.974530 2538 ts_tablet_manager.cc:1367] T 4c8e99e6b3a2407296d6fedfe6516947 P 88df0a071a0546aabeb57557af12bcff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:51.974684 2537 raft_consensus.cc:348] T 4c8e99e6b3a2407296d6fedfe6516947 P 661df3a5c18845898204ed4eab51c970 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } }
I20240502 13:59:51.974771 2537 raft_consensus.cc:374] T 4c8e99e6b3a2407296d6fedfe6516947 P 661df3a5c18845898204ed4eab51c970 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:51.974809 2537 raft_consensus.cc:729] T 4c8e99e6b3a2407296d6fedfe6516947 P 661df3a5c18845898204ed4eab51c970 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 661df3a5c18845898204ed4eab51c970, State: Initialized, Role: FOLLOWER
I20240502 13:59:51.974828 2538 raft_consensus.cc:348] T 4c8e99e6b3a2407296d6fedfe6516947 P 88df0a071a0546aabeb57557af12bcff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } }
I20240502 13:59:51.974922 2538 raft_consensus.cc:374] T 4c8e99e6b3a2407296d6fedfe6516947 P 88df0a071a0546aabeb57557af12bcff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:51.974974 2538 raft_consensus.cc:729] T 4c8e99e6b3a2407296d6fedfe6516947 P 88df0a071a0546aabeb57557af12bcff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88df0a071a0546aabeb57557af12bcff, State: Initialized, Role: FOLLOWER
I20240502 13:59:51.975042 2536 tablet_bootstrap.cc:492] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5: Bootstrap starting.
I20240502 13:59:51.975088 2538 consensus_queue.cc:260] T 4c8e99e6b3a2407296d6fedfe6516947 P 88df0a071a0546aabeb57557af12bcff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } }
I20240502 13:59:51.975387 2538 ts_tablet_manager.cc:1398] T 4c8e99e6b3a2407296d6fedfe6516947 P 88df0a071a0546aabeb57557af12bcff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:51.975522 2538 tablet_bootstrap.cc:492] T a0f5fad9037a43c686c6638e2bcb9ecb P 88df0a071a0546aabeb57557af12bcff: Bootstrap starting.
I20240502 13:59:51.975991 2536 tablet_bootstrap.cc:654] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:51.976250 2538 tablet_bootstrap.cc:654] T a0f5fad9037a43c686c6638e2bcb9ecb P 88df0a071a0546aabeb57557af12bcff: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:51.974925 2537 consensus_queue.cc:260] T 4c8e99e6b3a2407296d6fedfe6516947 P 661df3a5c18845898204ed4eab51c970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } }
I20240502 13:59:51.976898 2537 ts_tablet_manager.cc:1398] T 4c8e99e6b3a2407296d6fedfe6516947 P 661df3a5c18845898204ed4eab51c970: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 13:59:51.977085 2537 tablet_bootstrap.cc:492] T a0f5fad9037a43c686c6638e2bcb9ecb P 661df3a5c18845898204ed4eab51c970: Bootstrap starting.
I20240502 13:59:51.977221 2538 tablet_bootstrap.cc:492] T a0f5fad9037a43c686c6638e2bcb9ecb P 88df0a071a0546aabeb57557af12bcff: No bootstrap required, opened a new log
I20240502 13:59:51.977291 2538 ts_tablet_manager.cc:1367] T a0f5fad9037a43c686c6638e2bcb9ecb P 88df0a071a0546aabeb57557af12bcff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:51.977663 2538 raft_consensus.cc:348] T a0f5fad9037a43c686c6638e2bcb9ecb P 88df0a071a0546aabeb57557af12bcff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } } peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } }
I20240502 13:59:51.977759 2538 raft_consensus.cc:374] T a0f5fad9037a43c686c6638e2bcb9ecb P 88df0a071a0546aabeb57557af12bcff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:51.977802 2538 raft_consensus.cc:729] T a0f5fad9037a43c686c6638e2bcb9ecb P 88df0a071a0546aabeb57557af12bcff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88df0a071a0546aabeb57557af12bcff, State: Initialized, Role: FOLLOWER
I20240502 13:59:51.977859 2537 tablet_bootstrap.cc:654] T a0f5fad9037a43c686c6638e2bcb9ecb P 661df3a5c18845898204ed4eab51c970: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:51.977902 2538 consensus_queue.cc:260] T a0f5fad9037a43c686c6638e2bcb9ecb P 88df0a071a0546aabeb57557af12bcff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } } peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } }
I20240502 13:59:51.978158 2538 ts_tablet_manager.cc:1398] T a0f5fad9037a43c686c6638e2bcb9ecb P 88df0a071a0546aabeb57557af12bcff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:51.978773 2536 tablet_bootstrap.cc:492] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5: No bootstrap required, opened a new log
I20240502 13:59:51.978854 2536 ts_tablet_manager.cc:1367] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 13:59:51.979223 2537 tablet_bootstrap.cc:492] T a0f5fad9037a43c686c6638e2bcb9ecb P 661df3a5c18845898204ed4eab51c970: No bootstrap required, opened a new log
I20240502 13:59:51.979219 2536 raft_consensus.cc:348] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } } peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } }
I20240502 13:59:51.979336 2536 raft_consensus.cc:374] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:51.979347 2537 ts_tablet_manager.cc:1367] T a0f5fad9037a43c686c6638e2bcb9ecb P 661df3a5c18845898204ed4eab51c970: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 13:59:51.979384 2536 raft_consensus.cc:729] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a11d8cedf8848539f3de99c6a20ccc5, State: Initialized, Role: FOLLOWER
I20240502 13:59:51.979498 2536 consensus_queue.cc:260] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } } peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } }
I20240502 13:59:51.979733 2537 raft_consensus.cc:348] T a0f5fad9037a43c686c6638e2bcb9ecb P 661df3a5c18845898204ed4eab51c970 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } } peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } }
I20240502 13:59:51.979820 2536 ts_tablet_manager.cc:1398] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:51.979820 2537 raft_consensus.cc:374] T a0f5fad9037a43c686c6638e2bcb9ecb P 661df3a5c18845898204ed4eab51c970 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:51.979902 2537 raft_consensus.cc:729] T a0f5fad9037a43c686c6638e2bcb9ecb P 661df3a5c18845898204ed4eab51c970 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 661df3a5c18845898204ed4eab51c970, State: Initialized, Role: FOLLOWER
I20240502 13:59:51.980010 2537 consensus_queue.cc:260] T a0f5fad9037a43c686c6638e2bcb9ecb P 661df3a5c18845898204ed4eab51c970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } } peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } }
I20240502 13:59:51.980170 2536 tablet_bootstrap.cc:492] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5: Bootstrap starting.
I20240502 13:59:51.980262 2537 ts_tablet_manager.cc:1398] T a0f5fad9037a43c686c6638e2bcb9ecb P 661df3a5c18845898204ed4eab51c970: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:51.980942 2536 tablet_bootstrap.cc:654] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:51.981854 2536 tablet_bootstrap.cc:492] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5: No bootstrap required, opened a new log
I20240502 13:59:51.981940 2536 ts_tablet_manager.cc:1367] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 13:59:51.982340 2536 raft_consensus.cc:348] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } }
I20240502 13:59:51.982446 2536 raft_consensus.cc:374] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:51.982493 2536 raft_consensus.cc:729] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a11d8cedf8848539f3de99c6a20ccc5, State: Initialized, Role: FOLLOWER
I20240502 13:59:51.982707 2536 consensus_queue.cc:260] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } }
I20240502 13:59:51.982964 2536 ts_tablet_manager.cc:1398] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:51.992044 2546 raft_consensus.cc:482] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:51.992120 2546 raft_consensus.cc:504] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } } peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } }
I20240502 13:59:51.992440 2546 leader_election.cc:288] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 661df3a5c18845898204ed4eab51c970 (127.0.145.65:41099), 88df0a071a0546aabeb57557af12bcff (127.0.145.67:43723)
I20240502 13:59:51.996057 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0f5fad9037a43c686c6638e2bcb9ecb" candidate_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661df3a5c18845898204ed4eab51c970" is_pre_election: true
I20240502 13:59:51.996057 2489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0f5fad9037a43c686c6638e2bcb9ecb" candidate_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88df0a071a0546aabeb57557af12bcff" is_pre_election: true
I20240502 13:59:51.996204 2489 raft_consensus.cc:2454] T a0f5fad9037a43c686c6638e2bcb9ecb P 88df0a071a0546aabeb57557af12bcff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a11d8cedf8848539f3de99c6a20ccc5 in term 0.
I20240502 13:59:51.996204 2337 raft_consensus.cc:2454] T a0f5fad9037a43c686c6638e2bcb9ecb P 661df3a5c18845898204ed4eab51c970 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a11d8cedf8848539f3de99c6a20ccc5 in term 0.
I20240502 13:59:51.996485 2379 leader_election.cc:302] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a11d8cedf8848539f3de99c6a20ccc5, 88df0a071a0546aabeb57557af12bcff; no voters:
I20240502 13:59:51.996628 2546 raft_consensus.cc:2781] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:51.996685 2546 raft_consensus.cc:482] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:51.996731 2546 raft_consensus.cc:3037] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:52.032583 2546 raft_consensus.cc:504] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } } peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } }
I20240502 13:59:52.032980 2546 leader_election.cc:288] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5 [CANDIDATE]: Term 1 election: Requested vote from peers 661df3a5c18845898204ed4eab51c970 (127.0.145.65:41099), 88df0a071a0546aabeb57557af12bcff (127.0.145.67:43723)
I20240502 13:59:52.033286 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0f5fad9037a43c686c6638e2bcb9ecb" candidate_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661df3a5c18845898204ed4eab51c970"
I20240502 13:59:52.033286 2489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0f5fad9037a43c686c6638e2bcb9ecb" candidate_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88df0a071a0546aabeb57557af12bcff"
I20240502 13:59:52.033430 2337 raft_consensus.cc:3037] T a0f5fad9037a43c686c6638e2bcb9ecb P 661df3a5c18845898204ed4eab51c970 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:52.033433 2489 raft_consensus.cc:3037] T a0f5fad9037a43c686c6638e2bcb9ecb P 88df0a071a0546aabeb57557af12bcff [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:52.034386 2337 raft_consensus.cc:2454] T a0f5fad9037a43c686c6638e2bcb9ecb P 661df3a5c18845898204ed4eab51c970 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a11d8cedf8848539f3de99c6a20ccc5 in term 1.
I20240502 13:59:52.034386 2489 raft_consensus.cc:2454] T a0f5fad9037a43c686c6638e2bcb9ecb P 88df0a071a0546aabeb57557af12bcff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a11d8cedf8848539f3de99c6a20ccc5 in term 1.
I20240502 13:59:52.034662 2379 leader_election.cc:302] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a11d8cedf8848539f3de99c6a20ccc5, 88df0a071a0546aabeb57557af12bcff; no voters:
I20240502 13:59:52.034821 2546 raft_consensus.cc:2781] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:52.035040 2546 raft_consensus.cc:686] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5 [term 1 LEADER]: Becoming Leader. State: Replica: 6a11d8cedf8848539f3de99c6a20ccc5, State: Running, Role: LEADER
I20240502 13:59:52.035178 2546 consensus_queue.cc:237] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } } peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } }
I20240502 13:59:52.036499 2232 catalog_manager.cc:5484] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a11d8cedf8848539f3de99c6a20ccc5 (127.0.145.66). New cstate: current_term: 1 leader_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:52.173075 2546 raft_consensus.cc:482] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:52.173199 2546 raft_consensus.cc:504] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } }
I20240502 13:59:52.173528 2546 leader_election.cc:288] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 661df3a5c18845898204ed4eab51c970 (127.0.145.65:41099), 88df0a071a0546aabeb57557af12bcff (127.0.145.67:43723)
I20240502 13:59:52.173879 2489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8e99e6b3a2407296d6fedfe6516947" candidate_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88df0a071a0546aabeb57557af12bcff" is_pre_election: true
I20240502 13:59:52.173879 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8e99e6b3a2407296d6fedfe6516947" candidate_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661df3a5c18845898204ed4eab51c970" is_pre_election: true
I20240502 13:59:52.174060 2337 raft_consensus.cc:2454] T 4c8e99e6b3a2407296d6fedfe6516947 P 661df3a5c18845898204ed4eab51c970 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a11d8cedf8848539f3de99c6a20ccc5 in term 0.
I20240502 13:59:52.174060 2489 raft_consensus.cc:2454] T 4c8e99e6b3a2407296d6fedfe6516947 P 88df0a071a0546aabeb57557af12bcff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a11d8cedf8848539f3de99c6a20ccc5 in term 0.
I20240502 13:59:52.174305 2379 leader_election.cc:302] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 661df3a5c18845898204ed4eab51c970, 6a11d8cedf8848539f3de99c6a20ccc5; no voters:
I20240502 13:59:52.174453 2546 raft_consensus.cc:2781] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:52.174510 2546 raft_consensus.cc:482] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:52.174556 2546 raft_consensus.cc:3037] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:52.227730 2546 raft_consensus.cc:504] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } }
I20240502 13:59:52.228204 2546 leader_election.cc:288] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [CANDIDATE]: Term 1 election: Requested vote from peers 661df3a5c18845898204ed4eab51c970 (127.0.145.65:41099), 88df0a071a0546aabeb57557af12bcff (127.0.145.67:43723)
I20240502 13:59:52.228581 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8e99e6b3a2407296d6fedfe6516947" candidate_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661df3a5c18845898204ed4eab51c970"
I20240502 13:59:52.228621 2489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8e99e6b3a2407296d6fedfe6516947" candidate_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88df0a071a0546aabeb57557af12bcff"
I20240502 13:59:52.228729 2489 raft_consensus.cc:3037] T 4c8e99e6b3a2407296d6fedfe6516947 P 88df0a071a0546aabeb57557af12bcff [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:52.228726 2337 raft_consensus.cc:3037] T 4c8e99e6b3a2407296d6fedfe6516947 P 661df3a5c18845898204ed4eab51c970 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:52.229709 2337 raft_consensus.cc:2454] T 4c8e99e6b3a2407296d6fedfe6516947 P 661df3a5c18845898204ed4eab51c970 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a11d8cedf8848539f3de99c6a20ccc5 in term 1.
I20240502 13:59:52.229709 2489 raft_consensus.cc:2454] T 4c8e99e6b3a2407296d6fedfe6516947 P 88df0a071a0546aabeb57557af12bcff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a11d8cedf8848539f3de99c6a20ccc5 in term 1.
I20240502 13:59:52.230028 2379 leader_election.cc:302] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 661df3a5c18845898204ed4eab51c970, 6a11d8cedf8848539f3de99c6a20ccc5; no voters:
I20240502 13:59:52.230204 2546 raft_consensus.cc:2781] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:52.230314 2546 raft_consensus.cc:686] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [term 1 LEADER]: Becoming Leader. State: Replica: 6a11d8cedf8848539f3de99c6a20ccc5, State: Running, Role: LEADER
I20240502 13:59:52.230440 2546 consensus_queue.cc:237] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } }
I20240502 13:59:52.231863 2232 catalog_manager.cc:5484] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a11d8cedf8848539f3de99c6a20ccc5 (127.0.145.66). New cstate: current_term: 1 leader_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:52.536967 2546 consensus_queue.cc:1035] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:52.539815 2550 consensus_queue.cc:1035] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:52.636391 2550 consensus_queue.cc:1035] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:52.639783 2546 consensus_queue.cc:1035] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:52.772517 2288 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 13:59:52.776422 2414 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a0f5fad9037a43c686c6638e2bcb9ecb"
dest_uuid: "6a11d8cedf8848539f3de99c6a20ccc5"
mode: GRACEFUL
new_leader_uuid: "661df3a5c18845898204ed4eab51c970"
from {username='slave'} at 127.0.0.1:55668
I20240502 13:59:52.776548 2414 raft_consensus.cc:595] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5 [term 1 LEADER]: Received request to transfer leadership to TS 661df3a5c18845898204ed4eab51c970
I20240502 13:59:52.776893 2288 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240502 13:59:52.776973 2288 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 13:59:53.003716 2561 raft_consensus.cc:979] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5: : Instructing follower 661df3a5c18845898204ed4eab51c970 to start an election
I20240502 13:59:53.003832 2561 raft_consensus.cc:1067] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5 [term 1 LEADER]: Signalling peer 661df3a5c18845898204ed4eab51c970 to start an election
I20240502 13:59:53.004238 2337 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a0f5fad9037a43c686c6638e2bcb9ecb"
dest_uuid: "661df3a5c18845898204ed4eab51c970"
from {username='slave'} at 127.0.0.1:53412
I20240502 13:59:53.004366 2337 raft_consensus.cc:482] T a0f5fad9037a43c686c6638e2bcb9ecb P 661df3a5c18845898204ed4eab51c970 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 13:59:53.004419 2337 raft_consensus.cc:3037] T a0f5fad9037a43c686c6638e2bcb9ecb P 661df3a5c18845898204ed4eab51c970 [term 1 FOLLOWER]: Advancing to term 2
I20240502 13:59:53.005194 2337 raft_consensus.cc:504] T a0f5fad9037a43c686c6638e2bcb9ecb P 661df3a5c18845898204ed4eab51c970 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } } peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } }
I20240502 13:59:53.005597 2337 leader_election.cc:288] T a0f5fad9037a43c686c6638e2bcb9ecb P 661df3a5c18845898204ed4eab51c970 [CANDIDATE]: Term 2 election: Requested vote from peers 6a11d8cedf8848539f3de99c6a20ccc5 (127.0.145.66:41707), 88df0a071a0546aabeb57557af12bcff (127.0.145.67:43723)
I20240502 13:59:53.008677 2489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0f5fad9037a43c686c6638e2bcb9ecb" candidate_uuid: "661df3a5c18845898204ed4eab51c970" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "88df0a071a0546aabeb57557af12bcff"
I20240502 13:59:53.008693 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0f5fad9037a43c686c6638e2bcb9ecb" candidate_uuid: "661df3a5c18845898204ed4eab51c970" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6a11d8cedf8848539f3de99c6a20ccc5"
I20240502 13:59:53.008837 2489 raft_consensus.cc:3037] T a0f5fad9037a43c686c6638e2bcb9ecb P 88df0a071a0546aabeb57557af12bcff [term 1 FOLLOWER]: Advancing to term 2
I20240502 13:59:53.008841 2414 raft_consensus.cc:3032] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 13:59:53.008898 2414 raft_consensus.cc:729] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6a11d8cedf8848539f3de99c6a20ccc5, State: Running, Role: LEADER
I20240502 13:59:53.009045 2414 consensus_queue.cc:260] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } } peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } }
I20240502 13:59:53.009215 2414 raft_consensus.cc:3037] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 13:59:53.009790 2489 raft_consensus.cc:2454] T a0f5fad9037a43c686c6638e2bcb9ecb P 88df0a071a0546aabeb57557af12bcff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 661df3a5c18845898204ed4eab51c970 in term 2.
I20240502 13:59:53.010049 2414 raft_consensus.cc:2454] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 661df3a5c18845898204ed4eab51c970 in term 2.
I20240502 13:59:53.010087 2302 leader_election.cc:302] T a0f5fad9037a43c686c6638e2bcb9ecb P 661df3a5c18845898204ed4eab51c970 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 661df3a5c18845898204ed4eab51c970, 88df0a071a0546aabeb57557af12bcff; no voters:
I20240502 13:59:53.010257 2553 raft_consensus.cc:2781] T a0f5fad9037a43c686c6638e2bcb9ecb P 661df3a5c18845898204ed4eab51c970 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 13:59:53.010442 2553 raft_consensus.cc:686] T a0f5fad9037a43c686c6638e2bcb9ecb P 661df3a5c18845898204ed4eab51c970 [term 2 LEADER]: Becoming Leader. State: Replica: 661df3a5c18845898204ed4eab51c970, State: Running, Role: LEADER
I20240502 13:59:53.010563 2553 consensus_queue.cc:237] T a0f5fad9037a43c686c6638e2bcb9ecb P 661df3a5c18845898204ed4eab51c970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } } peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } }
I20240502 13:59:53.011759 2232 catalog_manager.cc:5484] T a0f5fad9037a43c686c6638e2bcb9ecb P 661df3a5c18845898204ed4eab51c970 reported cstate change: term changed from 1 to 2, leader changed from 6a11d8cedf8848539f3de99c6a20ccc5 (127.0.145.66) to 661df3a5c18845898204ed4eab51c970 (127.0.145.65). New cstate: current_term: 2 leader_uuid: "661df3a5c18845898204ed4eab51c970" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:53.558373 2414 raft_consensus.cc:1261] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5 [term 2 FOLLOWER]: Refusing update from remote peer 661df3a5c18845898204ed4eab51c970: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 13:59:53.558771 2572 consensus_queue.cc:1035] T a0f5fad9037a43c686c6638e2bcb9ecb P 661df3a5c18845898204ed4eab51c970 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 13:59:53.560810 2489 raft_consensus.cc:1261] T a0f5fad9037a43c686c6638e2bcb9ecb P 88df0a071a0546aabeb57557af12bcff [term 2 FOLLOWER]: Refusing update from remote peer 661df3a5c18845898204ed4eab51c970: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 13:59:53.561331 2572 consensus_queue.cc:1035] T a0f5fad9037a43c686c6638e2bcb9ecb P 661df3a5c18845898204ed4eab51c970 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 13:59:54.777215 2288 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 13:59:54.777549 2288 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 13:59:54.777607 2288 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 13:59:56.777839 2288 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 13:59:56.778154 2288 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 13:59:56.778209 2288 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 13:59:58.778450 2288 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 13:59:58.778753 2288 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 13:59:58.778805 2288 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:00:00.779098 2288 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:00:00.779635 2288 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:00:00.779700 2288 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:00:02.779958 2288 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:00:02.780321 2288 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:00:02.780387 2288 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:00:03.312933 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:03.315317 2588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:03.315892 581 server_base.cc:1034] running on GCE node
W20240502 14:00:03.316247 2586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:03.316653 2585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:03.316936 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:00:03.317013 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:00:03.317034 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658403317034 us; error 0 us; skew 500 ppm
I20240502 14:00:03.322489 581 webserver.cc:460] Webserver started at http://127.0.145.68:46385/ using document root <none> and password file <none>
I20240502 14:00:03.322651 581 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:03.322710 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:03.322788 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:03.323249 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658376531600-581-0/minicluster-data/ts-3-root/instance:
uuid: "00d9cfb0b7aa42cd97d1189f51fe1d89"
format_stamp: "Formatted at 2024-05-02 14:00:03 on dist-test-slave-1z5j"
I20240502 14:00:03.325006 581 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:03.326012 2593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:03.326268 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:03.326345 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658376531600-581-0/minicluster-data/ts-3-root
uuid: "00d9cfb0b7aa42cd97d1189f51fe1d89"
format_stamp: "Formatted at 2024-05-02 14:00:03 on dist-test-slave-1z5j"
I20240502 14:00:03.326413 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658376531600-581-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658376531600-581-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658376531600-581-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:03.342198 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:03.342540 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:03.342820 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:03.343309 581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:03.343390 581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:03.343461 581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:03.343494 581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:03.349123 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.68:43265
I20240502 14:00:03.349797 2657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.68:43265 every 8 connection(s)
I20240502 14:00:03.353713 2658 heartbeater.cc:347] Connected to a master server at 127.0.145.126:41589
I20240502 14:00:03.353790 2658 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:03.353981 2658 heartbeater.cc:511] Master 127.0.145.126:41589 requested a full tablet report, sending...
I20240502 14:00:03.354524 2232 ts_manager.cc:194] Registered new tserver with Master: 00d9cfb0b7aa42cd97d1189f51fe1d89 (127.0.145.68:43265)
I20240502 14:00:03.355016 2232 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55656
I20240502 14:00:03.786432 2414 consensus_queue.cc:237] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } attrs { replace: true } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } } peers { permanent_uuid: "00d9cfb0b7aa42cd97d1189f51fe1d89" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 43265 } attrs { promote: true } }
I20240502 14:00:03.787956 2337 raft_consensus.cc:1261] T 4c8e99e6b3a2407296d6fedfe6516947 P 661df3a5c18845898204ed4eab51c970 [term 1 FOLLOWER]: Refusing update from remote peer 6a11d8cedf8848539f3de99c6a20ccc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:03.787964 2489 raft_consensus.cc:1261] T 4c8e99e6b3a2407296d6fedfe6516947 P 88df0a071a0546aabeb57557af12bcff [term 1 FOLLOWER]: Refusing update from remote peer 6a11d8cedf8848539f3de99c6a20ccc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:03.788374 2666 consensus_queue.cc:1035] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:03.788578 2665 consensus_queue.cc:1035] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:03.790203 2665 raft_consensus.cc:2932] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 00d9cfb0b7aa42cd97d1189f51fe1d89 (127.0.145.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } attrs { replace: true } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } } peers { permanent_uuid: "00d9cfb0b7aa42cd97d1189f51fe1d89" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 43265 } attrs { promote: true } } }
I20240502 14:00:03.790664 2337 raft_consensus.cc:2932] T 4c8e99e6b3a2407296d6fedfe6516947 P 661df3a5c18845898204ed4eab51c970 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 00d9cfb0b7aa42cd97d1189f51fe1d89 (127.0.145.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } attrs { replace: true } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } } peers { permanent_uuid: "00d9cfb0b7aa42cd97d1189f51fe1d89" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 43265 } attrs { promote: true } } }
I20240502 14:00:03.790710 2489 raft_consensus.cc:2932] T 4c8e99e6b3a2407296d6fedfe6516947 P 88df0a071a0546aabeb57557af12bcff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 00d9cfb0b7aa42cd97d1189f51fe1d89 (127.0.145.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } attrs { replace: true } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } } peers { permanent_uuid: "00d9cfb0b7aa42cd97d1189f51fe1d89" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 43265 } attrs { promote: true } } }
I20240502 14:00:03.793157 2233 catalog_manager.cc:5484] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 reported cstate change: config changed from index -1 to 2, NON_VOTER 00d9cfb0b7aa42cd97d1189f51fe1d89 (127.0.145.68) added. New cstate: current_term: 1 leader_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00d9cfb0b7aa42cd97d1189f51fe1d89" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 43265 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:00:03.793992 2380 consensus_peers.cc:485] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 -> Peer 00d9cfb0b7aa42cd97d1189f51fe1d89 (127.0.145.68:43265): Couldn't send request to peer 00d9cfb0b7aa42cd97d1189f51fe1d89. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c8e99e6b3a2407296d6fedfe6516947. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:04.355749 2658 heartbeater.cc:503] Master 127.0.145.126:41589 was elected leader, sending a full tablet report...
I20240502 14:00:04.378535 2676 ts_tablet_manager.cc:902] T 4c8e99e6b3a2407296d6fedfe6516947 P 00d9cfb0b7aa42cd97d1189f51fe1d89: Initiating tablet copy from peer 6a11d8cedf8848539f3de99c6a20ccc5 (127.0.145.66:41707)
I20240502 14:00:04.379276 2676 tablet_copy_client.cc:308] T 4c8e99e6b3a2407296d6fedfe6516947 P 00d9cfb0b7aa42cd97d1189f51fe1d89: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.66:41707
I20240502 14:00:04.383075 2424 tablet_copy_service.cc:140] P 6a11d8cedf8848539f3de99c6a20ccc5: Received BeginTabletCopySession request for tablet 4c8e99e6b3a2407296d6fedfe6516947 from peer 00d9cfb0b7aa42cd97d1189f51fe1d89 ({username='slave'} at 127.0.0.1:49624)
I20240502 14:00:04.383200 2424 tablet_copy_service.cc:161] P 6a11d8cedf8848539f3de99c6a20ccc5: Beginning new tablet copy session on tablet 4c8e99e6b3a2407296d6fedfe6516947 from peer 00d9cfb0b7aa42cd97d1189f51fe1d89 at {username='slave'} at 127.0.0.1:49624: session id = 00d9cfb0b7aa42cd97d1189f51fe1d89-4c8e99e6b3a2407296d6fedfe6516947
I20240502 14:00:04.384639 2424 tablet_copy_source_session.cc:205] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:04.385933 2676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c8e99e6b3a2407296d6fedfe6516947. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:04.388042 2676 tablet_copy_client.cc:783] T 4c8e99e6b3a2407296d6fedfe6516947 P 00d9cfb0b7aa42cd97d1189f51fe1d89: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:04.388309 2676 tablet_copy_client.cc:647] T 4c8e99e6b3a2407296d6fedfe6516947 P 00d9cfb0b7aa42cd97d1189f51fe1d89: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:04.389600 2676 tablet_copy_client.cc:523] T 4c8e99e6b3a2407296d6fedfe6516947 P 00d9cfb0b7aa42cd97d1189f51fe1d89: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:04.390987 2676 tablet_bootstrap.cc:492] T 4c8e99e6b3a2407296d6fedfe6516947 P 00d9cfb0b7aa42cd97d1189f51fe1d89: Bootstrap starting.
I20240502 14:00:04.394785 2676 tablet_bootstrap.cc:492] T 4c8e99e6b3a2407296d6fedfe6516947 P 00d9cfb0b7aa42cd97d1189f51fe1d89: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:04.395004 2676 tablet_bootstrap.cc:492] T 4c8e99e6b3a2407296d6fedfe6516947 P 00d9cfb0b7aa42cd97d1189f51fe1d89: Bootstrap complete.
I20240502 14:00:04.395145 2676 ts_tablet_manager.cc:1367] T 4c8e99e6b3a2407296d6fedfe6516947 P 00d9cfb0b7aa42cd97d1189f51fe1d89: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 14:00:04.395503 2676 raft_consensus.cc:348] T 4c8e99e6b3a2407296d6fedfe6516947 P 00d9cfb0b7aa42cd97d1189f51fe1d89 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } attrs { replace: true } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } } peers { permanent_uuid: "00d9cfb0b7aa42cd97d1189f51fe1d89" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 43265 } attrs { promote: true } }
I20240502 14:00:04.395598 2676 raft_consensus.cc:729] T 4c8e99e6b3a2407296d6fedfe6516947 P 00d9cfb0b7aa42cd97d1189f51fe1d89 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 00d9cfb0b7aa42cd97d1189f51fe1d89, State: Initialized, Role: LEARNER
I20240502 14:00:04.395684 2676 consensus_queue.cc:260] T 4c8e99e6b3a2407296d6fedfe6516947 P 00d9cfb0b7aa42cd97d1189f51fe1d89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } attrs { replace: true } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } } peers { permanent_uuid: "00d9cfb0b7aa42cd97d1189f51fe1d89" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 43265 } attrs { promote: true } }
I20240502 14:00:04.396003 2676 ts_tablet_manager.cc:1398] T 4c8e99e6b3a2407296d6fedfe6516947 P 00d9cfb0b7aa42cd97d1189f51fe1d89: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:04.396474 2424 tablet_copy_service.cc:342] P 6a11d8cedf8848539f3de99c6a20ccc5: Request end of tablet copy session 00d9cfb0b7aa42cd97d1189f51fe1d89-4c8e99e6b3a2407296d6fedfe6516947 received from {username='slave'} at 127.0.0.1:49624
I20240502 14:00:04.396548 2424 tablet_copy_service.cc:433] P 6a11d8cedf8848539f3de99c6a20ccc5: ending tablet copy session 00d9cfb0b7aa42cd97d1189f51fe1d89-4c8e99e6b3a2407296d6fedfe6516947 on tablet 4c8e99e6b3a2407296d6fedfe6516947 with peer 00d9cfb0b7aa42cd97d1189f51fe1d89
I20240502 14:00:04.780630 2288 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:00:04.781029 2288 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:00:04.781119 2288 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:00:04.824906 2631 raft_consensus.cc:1203] T 4c8e99e6b3a2407296d6fedfe6516947 P 00d9cfb0b7aa42cd97d1189f51fe1d89 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:05.232535 2686 raft_consensus.cc:1050] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5: attempting to promote NON_VOTER 00d9cfb0b7aa42cd97d1189f51fe1d89 to VOTER
I20240502 14:00:05.233063 2686 consensus_queue.cc:237] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } attrs { replace: true } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } } peers { permanent_uuid: "00d9cfb0b7aa42cd97d1189f51fe1d89" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 43265 } attrs { promote: false } }
I20240502 14:00:05.234256 2631 raft_consensus.cc:1261] T 4c8e99e6b3a2407296d6fedfe6516947 P 00d9cfb0b7aa42cd97d1189f51fe1d89 [term 1 LEARNER]: Refusing update from remote peer 6a11d8cedf8848539f3de99c6a20ccc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:05.234311 2337 raft_consensus.cc:1261] T 4c8e99e6b3a2407296d6fedfe6516947 P 661df3a5c18845898204ed4eab51c970 [term 1 FOLLOWER]: Refusing update from remote peer 6a11d8cedf8848539f3de99c6a20ccc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:05.234424 2489 raft_consensus.cc:1261] T 4c8e99e6b3a2407296d6fedfe6516947 P 88df0a071a0546aabeb57557af12bcff [term 1 FOLLOWER]: Refusing update from remote peer 6a11d8cedf8848539f3de99c6a20ccc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:05.234623 2685 consensus_queue.cc:1035] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:05.234786 2686 consensus_queue.cc:1035] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:05.234969 2688 consensus_queue.cc:1035] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00d9cfb0b7aa42cd97d1189f51fe1d89" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 43265 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:05.237104 2666 raft_consensus.cc:2932] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 00d9cfb0b7aa42cd97d1189f51fe1d89 (127.0.145.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } attrs { replace: true } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } } peers { permanent_uuid: "00d9cfb0b7aa42cd97d1189f51fe1d89" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 43265 } attrs { promote: false } } }
I20240502 14:00:05.238610 2631 raft_consensus.cc:2932] T 4c8e99e6b3a2407296d6fedfe6516947 P 00d9cfb0b7aa42cd97d1189f51fe1d89 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 00d9cfb0b7aa42cd97d1189f51fe1d89 (127.0.145.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } attrs { replace: true } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } } peers { permanent_uuid: "00d9cfb0b7aa42cd97d1189f51fe1d89" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 43265 } attrs { promote: false } } }
I20240502 14:00:05.239012 2489 raft_consensus.cc:2932] T 4c8e99e6b3a2407296d6fedfe6516947 P 88df0a071a0546aabeb57557af12bcff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 00d9cfb0b7aa42cd97d1189f51fe1d89 (127.0.145.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } attrs { replace: true } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } } peers { permanent_uuid: "00d9cfb0b7aa42cd97d1189f51fe1d89" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 43265 } attrs { promote: false } } }
I20240502 14:00:05.239063 2337 raft_consensus.cc:2932] T 4c8e99e6b3a2407296d6fedfe6516947 P 661df3a5c18845898204ed4eab51c970 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 00d9cfb0b7aa42cd97d1189f51fe1d89 (127.0.145.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } attrs { replace: true } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } } peers { permanent_uuid: "00d9cfb0b7aa42cd97d1189f51fe1d89" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 43265 } attrs { promote: false } } }
I20240502 14:00:05.239356 2233 catalog_manager.cc:5484] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 reported cstate change: config changed from index 2 to 3, 00d9cfb0b7aa42cd97d1189f51fe1d89 (127.0.145.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "661df3a5c18845898204ed4eab51c970" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 41099 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00d9cfb0b7aa42cd97d1189f51fe1d89" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 43265 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:05.244067 2414 consensus_queue.cc:237] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } } peers { permanent_uuid: "00d9cfb0b7aa42cd97d1189f51fe1d89" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 43265 } attrs { promote: false } }
I20240502 14:00:05.244997 2631 raft_consensus.cc:1261] T 4c8e99e6b3a2407296d6fedfe6516947 P 00d9cfb0b7aa42cd97d1189f51fe1d89 [term 1 FOLLOWER]: Refusing update from remote peer 6a11d8cedf8848539f3de99c6a20ccc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:05.244997 2489 raft_consensus.cc:1261] T 4c8e99e6b3a2407296d6fedfe6516947 P 88df0a071a0546aabeb57557af12bcff [term 1 FOLLOWER]: Refusing update from remote peer 6a11d8cedf8848539f3de99c6a20ccc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:05.245337 2685 consensus_queue.cc:1035] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00d9cfb0b7aa42cd97d1189f51fe1d89" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 43265 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:05.245525 2686 consensus_queue.cc:1035] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:05.246958 2688 raft_consensus.cc:2932] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 661df3a5c18845898204ed4eab51c970 (127.0.145.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } } peers { permanent_uuid: "00d9cfb0b7aa42cd97d1189f51fe1d89" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 43265 } attrs { promote: false } } }
I20240502 14:00:05.247249 2631 raft_consensus.cc:2932] T 4c8e99e6b3a2407296d6fedfe6516947 P 00d9cfb0b7aa42cd97d1189f51fe1d89 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 661df3a5c18845898204ed4eab51c970 (127.0.145.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } } peers { permanent_uuid: "00d9cfb0b7aa42cd97d1189f51fe1d89" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 43265 } attrs { promote: false } } }
I20240502 14:00:05.247416 2489 raft_consensus.cc:2932] T 4c8e99e6b3a2407296d6fedfe6516947 P 88df0a071a0546aabeb57557af12bcff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 661df3a5c18845898204ed4eab51c970 (127.0.145.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } } peers { permanent_uuid: "00d9cfb0b7aa42cd97d1189f51fe1d89" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 43265 } attrs { promote: false } } }
I20240502 14:00:05.248914 2218 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4c8e99e6b3a2407296d6fedfe6516947 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:05.249161 2232 catalog_manager.cc:5484] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 reported cstate change: config changed from index 3 to 4, VOTER 661df3a5c18845898204ed4eab51c970 (127.0.145.65) evicted. New cstate: current_term: 1 leader_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a11d8cedf8848539f3de99c6a20ccc5" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88df0a071a0546aabeb57557af12bcff" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 43723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00d9cfb0b7aa42cd97d1189f51fe1d89" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 43265 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:05.251152 2327 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c8e99e6b3a2407296d6fedfe6516947 with delete_type TABLET_DATA_TOMBSTONED (TS 661df3a5c18845898204ed4eab51c970 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53406
I20240502 14:00:05.251732 2697 tablet_replica.cc:330] T 4c8e99e6b3a2407296d6fedfe6516947 P 661df3a5c18845898204ed4eab51c970: stopping tablet replica
I20240502 14:00:05.251900 2697 raft_consensus.cc:2229] T 4c8e99e6b3a2407296d6fedfe6516947 P 661df3a5c18845898204ed4eab51c970 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:05.252024 2697 raft_consensus.cc:2258] T 4c8e99e6b3a2407296d6fedfe6516947 P 661df3a5c18845898204ed4eab51c970 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:05.252560 2697 ts_tablet_manager.cc:1875] T 4c8e99e6b3a2407296d6fedfe6516947 P 661df3a5c18845898204ed4eab51c970: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:05.254179 2697 ts_tablet_manager.cc:1888] T 4c8e99e6b3a2407296d6fedfe6516947 P 661df3a5c18845898204ed4eab51c970: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:00:05.254247 2697 log.cc:1196] T 4c8e99e6b3a2407296d6fedfe6516947 P 661df3a5c18845898204ed4eab51c970: Deleting WAL directory at /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658376531600-581-0/minicluster-data/ts-0-root/wals/4c8e99e6b3a2407296d6fedfe6516947
I20240502 14:00:05.254686 2219 catalog_manager.cc:4830] TS 661df3a5c18845898204ed4eab51c970 (127.0.145.65:41099): tablet 4c8e99e6b3a2407296d6fedfe6516947 (table test-workload [id=110b644d079c43f5a2dadd853b562fdd]) successfully deleted
I20240502 14:00:06.781369 2288 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:00:06.781710 2288 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:00:06.781766 2288 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:00:08.782004 2288 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:00:08.782341 2288 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:00:08.782397 2288 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:00:10.782694 2288 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:00:10.783066 2288 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:00:10.783125 2288 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:00:12.783411 2288 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:00:12.783763 2288 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:00:12.783823 2288 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:00:14.784132 2288 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:00:14.784495 2288 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:00:14.784551 2288 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:00:16.784879 2288 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:00:16.785243 2288 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:00:16.785296 2288 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:00:16.942699 581 tablet_server.cc:178] TabletServer@127.0.145.65:0 shutting down...
I20240502 14:00:16.946281 581 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:00:16.946471 581 tablet_replica.cc:330] T a0f5fad9037a43c686c6638e2bcb9ecb P 661df3a5c18845898204ed4eab51c970: stopping tablet replica
I20240502 14:00:16.946605 581 raft_consensus.cc:2229] T a0f5fad9037a43c686c6638e2bcb9ecb P 661df3a5c18845898204ed4eab51c970 [term 2 LEADER]: Raft consensus shutting down.
I20240502 14:00:16.946851 581 raft_consensus.cc:2258] T a0f5fad9037a43c686c6638e2bcb9ecb P 661df3a5c18845898204ed4eab51c970 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:16.960014 581 tablet_server.cc:194] TabletServer@127.0.145.65:0 shutdown complete.
I20240502 14:00:16.961881 581 tablet_server.cc:178] TabletServer@127.0.145.66:0 shutting down...
I20240502 14:00:16.965680 581 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:00:16.965824 581 tablet_replica.cc:330] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5: stopping tablet replica
I20240502 14:00:16.965927 581 raft_consensus.cc:2229] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:16.966080 581 raft_consensus.cc:2258] T 4c8e99e6b3a2407296d6fedfe6516947 P 6a11d8cedf8848539f3de99c6a20ccc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:16.966424 581 tablet_replica.cc:330] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5: stopping tablet replica
I20240502 14:00:16.966491 581 raft_consensus.cc:2229] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:16.966563 581 raft_consensus.cc:2258] T a0f5fad9037a43c686c6638e2bcb9ecb P 6a11d8cedf8848539f3de99c6a20ccc5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:16.978816 581 tablet_server.cc:194] TabletServer@127.0.145.66:0 shutdown complete.
I20240502 14:00:16.980439 581 tablet_server.cc:178] TabletServer@127.0.145.67:0 shutting down...
I20240502 14:00:16.984063 581 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:00:16.984200 581 tablet_replica.cc:330] T a0f5fad9037a43c686c6638e2bcb9ecb P 88df0a071a0546aabeb57557af12bcff: stopping tablet replica
I20240502 14:00:16.984287 581 raft_consensus.cc:2229] T a0f5fad9037a43c686c6638e2bcb9ecb P 88df0a071a0546aabeb57557af12bcff [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:16.984376 581 raft_consensus.cc:2258] T a0f5fad9037a43c686c6638e2bcb9ecb P 88df0a071a0546aabeb57557af12bcff [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:16.984683 581 tablet_replica.cc:330] T 4c8e99e6b3a2407296d6fedfe6516947 P 88df0a071a0546aabeb57557af12bcff: stopping tablet replica
I20240502 14:00:16.984776 581 raft_consensus.cc:2229] T 4c8e99e6b3a2407296d6fedfe6516947 P 88df0a071a0546aabeb57557af12bcff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:16.984886 581 raft_consensus.cc:2258] T 4c8e99e6b3a2407296d6fedfe6516947 P 88df0a071a0546aabeb57557af12bcff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:16.997710 581 tablet_server.cc:194] TabletServer@127.0.145.67:0 shutdown complete.
I20240502 14:00:16.999253 581 tablet_server.cc:178] TabletServer@127.0.145.68:0 shutting down...
I20240502 14:00:17.003050 581 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:00:17.003193 581 tablet_replica.cc:330] T 4c8e99e6b3a2407296d6fedfe6516947 P 00d9cfb0b7aa42cd97d1189f51fe1d89: stopping tablet replica
I20240502 14:00:17.003291 581 raft_consensus.cc:2229] T 4c8e99e6b3a2407296d6fedfe6516947 P 00d9cfb0b7aa42cd97d1189f51fe1d89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:17.003382 581 raft_consensus.cc:2258] T 4c8e99e6b3a2407296d6fedfe6516947 P 00d9cfb0b7aa42cd97d1189f51fe1d89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:17.015908 581 tablet_server.cc:194] TabletServer@127.0.145.68:0 shutdown complete.
I20240502 14:00:17.017524 581 master.cc:537] Master@127.0.145.126:41589 shutting down...
I20240502 14:00:17.020424 581 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 468d6d29941242bdab578cd9b29ccf2f [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:17.020529 581 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 468d6d29941242bdab578cd9b29ccf2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:17.020572 581 tablet_replica.cc:330] T 00000000000000000000000000000000 P 468d6d29941242bdab578cd9b29ccf2f: stopping tablet replica
I20240502 14:00:17.032426 581 master.cc:552] Master@127.0.145.126:41589 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (26310 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20240502 14:00:17.038678 581 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.145.126:38285
I20240502 14:00:17.038951 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:17.040825 2703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:17.041365 581 server_base.cc:1034] running on GCE node
W20240502 14:00:17.041810 2706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:17.042230 2704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:17.042395 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:00:17.042444 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:00:17.042464 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658417042463 us; error 0 us; skew 500 ppm
I20240502 14:00:17.043231 581 webserver.cc:460] Webserver started at http://127.0.145.126:35137/ using document root <none> and password file <none>
I20240502 14:00:17.043351 581 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:17.043397 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:17.043474 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:17.043802 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714658376531600-581-0/minicluster-data/master-0-root/instance:
uuid: "a455973b1f724c68891ddd0c340fbe24"
format_stamp: "Formatted at 2024-05-02 14:00:17 on dist-test-slave-1z5j"
I20240502 14:00:17.045089 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:17.045878 2711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:17.046082 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:17.046142 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714658376531600-581-0/minicluster-data/master-0-root
uuid: "a455973b1f724c68891ddd0c340fbe24"
format_stamp: "Formatted at 2024-05-02 14:00:17 on dist-test-slave-1z5j"
I20240502 14:00:17.046203 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714658376531600-581-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714658376531600-581-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714658376531600-581-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:17.056819 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:17.057072 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:17.060623 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.126:38285
I20240502 14:00:17.062608 2763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.126:38285 every 8 connection(s)
I20240502 14:00:17.063001 2764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:17.064555 2764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a455973b1f724c68891ddd0c340fbe24: Bootstrap starting.
I20240502 14:00:17.065290 2764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a455973b1f724c68891ddd0c340fbe24: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:17.066135 2764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a455973b1f724c68891ddd0c340fbe24: No bootstrap required, opened a new log
I20240502 14:00:17.066453 2764 raft_consensus.cc:348] T 00000000000000000000000000000000 P a455973b1f724c68891ddd0c340fbe24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a455973b1f724c68891ddd0c340fbe24" member_type: VOTER }
I20240502 14:00:17.066530 2764 raft_consensus.cc:374] T 00000000000000000000000000000000 P a455973b1f724c68891ddd0c340fbe24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:17.066556 2764 raft_consensus.cc:729] T 00000000000000000000000000000000 P a455973b1f724c68891ddd0c340fbe24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a455973b1f724c68891ddd0c340fbe24, State: Initialized, Role: FOLLOWER
I20240502 14:00:17.066658 2764 consensus_queue.cc:260] T 00000000000000000000000000000000 P a455973b1f724c68891ddd0c340fbe24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a455973b1f724c68891ddd0c340fbe24" member_type: VOTER }
I20240502 14:00:17.066727 2764 raft_consensus.cc:388] T 00000000000000000000000000000000 P a455973b1f724c68891ddd0c340fbe24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:00:17.066756 2764 raft_consensus.cc:482] T 00000000000000000000000000000000 P a455973b1f724c68891ddd0c340fbe24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:00:17.066803 2764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a455973b1f724c68891ddd0c340fbe24 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:17.067356 2764 raft_consensus.cc:504] T 00000000000000000000000000000000 P a455973b1f724c68891ddd0c340fbe24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a455973b1f724c68891ddd0c340fbe24" member_type: VOTER }
I20240502 14:00:17.067464 2764 leader_election.cc:302] T 00000000000000000000000000000000 P a455973b1f724c68891ddd0c340fbe24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a455973b1f724c68891ddd0c340fbe24; no voters:
I20240502 14:00:17.067596 2764 leader_election.cc:288] T 00000000000000000000000000000000 P a455973b1f724c68891ddd0c340fbe24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:00:17.067700 2767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a455973b1f724c68891ddd0c340fbe24 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:17.067884 2767 raft_consensus.cc:686] T 00000000000000000000000000000000 P a455973b1f724c68891ddd0c340fbe24 [term 1 LEADER]: Becoming Leader. State: Replica: a455973b1f724c68891ddd0c340fbe24, State: Running, Role: LEADER
I20240502 14:00:17.067972 2764 sys_catalog.cc:564] T 00000000000000000000000000000000 P a455973b1f724c68891ddd0c340fbe24 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:00:17.068012 2767 consensus_queue.cc:237] T 00000000000000000000000000000000 P a455973b1f724c68891ddd0c340fbe24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a455973b1f724c68891ddd0c340fbe24" member_type: VOTER }
I20240502 14:00:17.068393 2769 sys_catalog.cc:455] T 00000000000000000000000000000000 P a455973b1f724c68891ddd0c340fbe24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a455973b1f724c68891ddd0c340fbe24. Latest consensus state: current_term: 1 leader_uuid: "a455973b1f724c68891ddd0c340fbe24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a455973b1f724c68891ddd0c340fbe24" member_type: VOTER } }
I20240502 14:00:17.068379 2768 sys_catalog.cc:455] T 00000000000000000000000000000000 P a455973b1f724c68891ddd0c340fbe24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a455973b1f724c68891ddd0c340fbe24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a455973b1f724c68891ddd0c340fbe24" member_type: VOTER } }
I20240502 14:00:17.068513 2769 sys_catalog.cc:458] T 00000000000000000000000000000000 P a455973b1f724c68891ddd0c340fbe24 [sys.catalog]: This master's current role is: LEADER
I20240502 14:00:17.068540 2768 sys_catalog.cc:458] T 00000000000000000000000000000000 P a455973b1f724c68891ddd0c340fbe24 [sys.catalog]: This master's current role is: LEADER
I20240502 14:00:17.068779 2773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:00:17.069526 2773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:00:17.069770 581 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 14:00:17.071107 2773 catalog_manager.cc:1347] Generated new cluster ID: 2c10b59154e244e09593f9f745517b3f
I20240502 14:00:17.071152 2773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:00:17.076622 2773 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:00:17.077104 2773 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:00:17.085037 2773 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a455973b1f724c68891ddd0c340fbe24: Generated new TSK 0
I20240502 14:00:17.085160 2773 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:00:17.101495 581 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 4.9368e-05s
I20240502 14:00:29.177312 581 master.cc:537] Master@127.0.145.126:38285 shutting down...
I20240502 14:00:29.180665 581 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a455973b1f724c68891ddd0c340fbe24 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:29.180871 581 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a455973b1f724c68891ddd0c340fbe24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:29.180949 581 tablet_replica.cc:330] T 00000000000000000000000000000000 P a455973b1f724c68891ddd0c340fbe24: stopping tablet replica
I20240502 14:00:29.192747 581 master.cc:552] Master@127.0.145.126:38285 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12157 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20240502 14:00:29.196336 581 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.145.126:38739
I20240502 14:00:29.196602 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:29.198446 2787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:29.198751 2786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:29.199218 2789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:29.199594 581 server_base.cc:1034] running on GCE node
I20240502 14:00:29.199716 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:00:29.199749 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:00:29.199769 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658429199768 us; error 0 us; skew 500 ppm
I20240502 14:00:29.206655 581 webserver.cc:460] Webserver started at http://127.0.145.126:37527/ using document root <none> and password file <none>
I20240502 14:00:29.206825 581 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:29.206876 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:29.206950 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:29.207307 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658376531600-581-0/minicluster-data/master-0-root/instance:
uuid: "5c9aa9dd036b48bd9fbfd0b502576d67"
format_stamp: "Formatted at 2024-05-02 14:00:29 on dist-test-slave-1z5j"
I20240502 14:00:29.208640 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:29.209587 2794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:29.209800 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:29.209885 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658376531600-581-0/minicluster-data/master-0-root
uuid: "5c9aa9dd036b48bd9fbfd0b502576d67"
format_stamp: "Formatted at 2024-05-02 14:00:29 on dist-test-slave-1z5j"
I20240502 14:00:29.209952 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658376531600-581-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658376531600-581-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658376531600-581-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:29.246009 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:29.246376 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:29.250063 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.126:38739
I20240502 14:00:29.250473 2847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.126:38739 every 8 connection(s)
I20240502 14:00:29.250989 2848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.252673 2848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9aa9dd036b48bd9fbfd0b502576d67: Bootstrap starting.
I20240502 14:00:29.253396 2848 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c9aa9dd036b48bd9fbfd0b502576d67: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.254266 2848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9aa9dd036b48bd9fbfd0b502576d67: No bootstrap required, opened a new log
I20240502 14:00:29.254595 2848 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c9aa9dd036b48bd9fbfd0b502576d67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9aa9dd036b48bd9fbfd0b502576d67" member_type: VOTER }
I20240502 14:00:29.254678 2848 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c9aa9dd036b48bd9fbfd0b502576d67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.254705 2848 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c9aa9dd036b48bd9fbfd0b502576d67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c9aa9dd036b48bd9fbfd0b502576d67, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.254829 2848 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c9aa9dd036b48bd9fbfd0b502576d67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9aa9dd036b48bd9fbfd0b502576d67" member_type: VOTER }
I20240502 14:00:29.254892 2848 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c9aa9dd036b48bd9fbfd0b502576d67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:00:29.254922 2848 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c9aa9dd036b48bd9fbfd0b502576d67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:00:29.254974 2848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c9aa9dd036b48bd9fbfd0b502576d67 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.255566 2848 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c9aa9dd036b48bd9fbfd0b502576d67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9aa9dd036b48bd9fbfd0b502576d67" member_type: VOTER }
I20240502 14:00:29.255677 2848 leader_election.cc:302] T 00000000000000000000000000000000 P 5c9aa9dd036b48bd9fbfd0b502576d67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c9aa9dd036b48bd9fbfd0b502576d67; no voters:
I20240502 14:00:29.255826 2848 leader_election.cc:288] T 00000000000000000000000000000000 P 5c9aa9dd036b48bd9fbfd0b502576d67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:00:29.255940 2851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c9aa9dd036b48bd9fbfd0b502576d67 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:29.256124 2851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c9aa9dd036b48bd9fbfd0b502576d67 [term 1 LEADER]: Becoming Leader. State: Replica: 5c9aa9dd036b48bd9fbfd0b502576d67, State: Running, Role: LEADER
I20240502 14:00:29.256259 2848 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c9aa9dd036b48bd9fbfd0b502576d67 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:00:29.256249 2851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c9aa9dd036b48bd9fbfd0b502576d67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9aa9dd036b48bd9fbfd0b502576d67" member_type: VOTER }
I20240502 14:00:29.256693 2852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9aa9dd036b48bd9fbfd0b502576d67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c9aa9dd036b48bd9fbfd0b502576d67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9aa9dd036b48bd9fbfd0b502576d67" member_type: VOTER } }
I20240502 14:00:29.256721 2853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9aa9dd036b48bd9fbfd0b502576d67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c9aa9dd036b48bd9fbfd0b502576d67. Latest consensus state: current_term: 1 leader_uuid: "5c9aa9dd036b48bd9fbfd0b502576d67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9aa9dd036b48bd9fbfd0b502576d67" member_type: VOTER } }
I20240502 14:00:29.256951 2853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9aa9dd036b48bd9fbfd0b502576d67 [sys.catalog]: This master's current role is: LEADER
I20240502 14:00:29.256865 2852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9aa9dd036b48bd9fbfd0b502576d67 [sys.catalog]: This master's current role is: LEADER
I20240502 14:00:29.257479 2859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:00:29.258288 2859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:00:29.258446 581 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 14:00:29.259927 2859 catalog_manager.cc:1347] Generated new cluster ID: 46739eb6f70a4dae89bf20ad0ad9d53e
I20240502 14:00:29.259972 2859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:00:29.269223 2859 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:00:29.269695 2859 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:00:29.275678 2859 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5c9aa9dd036b48bd9fbfd0b502576d67: Generated new TSK 0
I20240502 14:00:29.275810 2859 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:00:29.290458 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:29.292555 2870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:29.292852 2869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:29.293316 2872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:29.293807 581 server_base.cc:1034] running on GCE node
I20240502 14:00:29.293941 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:00:29.293977 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:00:29.293989 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658429293989 us; error 0 us; skew 500 ppm
I20240502 14:00:29.294778 581 webserver.cc:460] Webserver started at http://127.0.145.65:34481/ using document root <none> and password file <none>
I20240502 14:00:29.294899 581 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:29.294951 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:29.295028 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:29.295362 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658376531600-581-0/minicluster-data/ts-0-root/instance:
uuid: "edf3fb18ae134d219f4987d160b3635e"
format_stamp: "Formatted at 2024-05-02 14:00:29 on dist-test-slave-1z5j"
I20240502 14:00:29.296600 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:29.297416 2877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:29.297628 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:29.297689 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658376531600-581-0/minicluster-data/ts-0-root
uuid: "edf3fb18ae134d219f4987d160b3635e"
format_stamp: "Formatted at 2024-05-02 14:00:29 on dist-test-slave-1z5j"
I20240502 14:00:29.297748 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658376531600-581-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658376531600-581-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658376531600-581-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:29.302690 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:29.302948 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:29.303167 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:29.303575 581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:29.303664 581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:29.303745 581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:29.303776 581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:29.307410 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.65:43547
I20240502 14:00:29.307428 2940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.65:43547 every 8 connection(s)
I20240502 14:00:29.308081 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:29.309953 2946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:29.310425 2945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:29.310715 2948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:29.311136 581 server_base.cc:1034] running on GCE node
I20240502 14:00:29.311303 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:00:29.311350 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:00:29.311378 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658429311377 us; error 0 us; skew 500 ppm
I20240502 14:00:29.312091 581 webserver.cc:460] Webserver started at http://127.0.145.66:39897/ using document root <none> and password file <none>
I20240502 14:00:29.312232 581 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:29.312280 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:29.312331 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:29.312595 2941 heartbeater.cc:347] Connected to a master server at 127.0.145.126:38739
I20240502 14:00:29.312686 2941 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:29.312705 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658376531600-581-0/minicluster-data/ts-1-root/instance:
uuid: "badc4cffac9944b2a6537d91ef1ac2c7"
format_stamp: "Formatted at 2024-05-02 14:00:29 on dist-test-slave-1z5j"
I20240502 14:00:29.312911 2941 heartbeater.cc:511] Master 127.0.145.126:38739 requested a full tablet report, sending...
I20240502 14:00:29.313519 2811 ts_manager.cc:194] Registered new tserver with Master: edf3fb18ae134d219f4987d160b3635e (127.0.145.65:43547)
I20240502 14:00:29.314204 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:29.314420 2811 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43172
I20240502 14:00:29.315090 2953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:29.315302 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:29.315358 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658376531600-581-0/minicluster-data/ts-1-root
uuid: "badc4cffac9944b2a6537d91ef1ac2c7"
format_stamp: "Formatted at 2024-05-02 14:00:29 on dist-test-slave-1z5j"
I20240502 14:00:29.315405 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658376531600-581-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658376531600-581-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658376531600-581-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:29.321753 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:29.321977 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:29.322166 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:29.322491 581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:29.322525 581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:29.322551 581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:29.322568 581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:29.326328 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.66:41247
I20240502 14:00:29.326366 3017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.66:41247 every 8 connection(s)
I20240502 14:00:29.326999 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:29.328601 3021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:29.329092 3022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:29.329560 3024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:29.329881 581 server_base.cc:1034] running on GCE node
I20240502 14:00:29.330055 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:00:29.330101 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:00:29.330119 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658429330119 us; error 0 us; skew 500 ppm
I20240502 14:00:29.330725 3018 heartbeater.cc:347] Connected to a master server at 127.0.145.126:38739
I20240502 14:00:29.330799 3018 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:29.330981 3018 heartbeater.cc:511] Master 127.0.145.126:38739 requested a full tablet report, sending...
I20240502 14:00:29.331007 581 webserver.cc:460] Webserver started at http://127.0.145.67:39203/ using document root <none> and password file <none>
I20240502 14:00:29.331132 581 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:29.331182 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:29.331239 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:29.331516 2811 ts_manager.cc:194] Registered new tserver with Master: badc4cffac9944b2a6537d91ef1ac2c7 (127.0.145.66:41247)
I20240502 14:00:29.331616 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658376531600-581-0/minicluster-data/ts-2-root/instance:
uuid: "f2ab283b75814f8fbe6e45292f715057"
format_stamp: "Formatted at 2024-05-02 14:00:29 on dist-test-slave-1z5j"
I20240502 14:00:29.332144 2811 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43180
I20240502 14:00:29.333060 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:29.333793 3029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:29.333990 581 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240502 14:00:29.334049 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658376531600-581-0/minicluster-data/ts-2-root
uuid: "f2ab283b75814f8fbe6e45292f715057"
format_stamp: "Formatted at 2024-05-02 14:00:29 on dist-test-slave-1z5j"
I20240502 14:00:29.334106 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658376531600-581-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658376531600-581-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658376531600-581-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:29.340394 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:29.340672 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:29.340938 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:29.341310 581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:29.341346 581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:29.341380 581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:29.341408 581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:29.345372 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.67:46817
I20240502 14:00:29.345418 3093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.67:46817 every 8 connection(s)
I20240502 14:00:29.348909 3094 heartbeater.cc:347] Connected to a master server at 127.0.145.126:38739
I20240502 14:00:29.348980 3094 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:29.349143 3094 heartbeater.cc:511] Master 127.0.145.126:38739 requested a full tablet report, sending...
I20240502 14:00:29.349632 2811 ts_manager.cc:194] Registered new tserver with Master: f2ab283b75814f8fbe6e45292f715057 (127.0.145.67:46817)
I20240502 14:00:29.350069 2811 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43192
I20240502 14:00:29.350170 581 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.00444538s
I20240502 14:00:30.315250 2941 heartbeater.cc:503] Master 127.0.145.126:38739 was elected leader, sending a full tablet report...
I20240502 14:00:30.332866 3018 heartbeater.cc:503] Master 127.0.145.126:38739 was elected leader, sending a full tablet report...
I20240502 14:00:30.350849 3094 heartbeater.cc:503] Master 127.0.145.126:38739 was elected leader, sending a full tablet report...
I20240502 14:00:31.258040 2866 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:00:33.258397 2866 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:00:35.258714 2866 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:00:37.259034 2866 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:00:39.259349 2866 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:00:41.259670 2866 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:00:41.425748 581 tablet_server.cc:178] TabletServer@127.0.145.65:0 shutting down...
I20240502 14:00:41.429023 581 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:00:41.440659 581 tablet_server.cc:194] TabletServer@127.0.145.65:0 shutdown complete.
I20240502 14:00:41.442306 581 tablet_server.cc:178] TabletServer@127.0.145.66:0 shutting down...
I20240502 14:00:41.445412 581 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:00:41.457069 581 tablet_server.cc:194] TabletServer@127.0.145.66:0 shutdown complete.
I20240502 14:00:41.458545 581 tablet_server.cc:178] TabletServer@127.0.145.67:0 shutting down...
I20240502 14:00:41.461422 581 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:00:41.472851 581 tablet_server.cc:194] TabletServer@127.0.145.67:0 shutdown complete.
I20240502 14:00:41.474195 581 master.cc:537] Master@127.0.145.126:38739 shutting down...
I20240502 14:00:41.476963 581 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5c9aa9dd036b48bd9fbfd0b502576d67 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:41.477093 581 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5c9aa9dd036b48bd9fbfd0b502576d67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:41.477141 581 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5c9aa9dd036b48bd9fbfd0b502576d67: stopping tablet replica
I20240502 14:00:41.488814 581 master.cc:552] Master@127.0.145.126:38739 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12297 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20240502 14:00:41.493513 581 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.145.126:36291
I20240502 14:00:41.493796 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:41.495648 3098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:41.496218 581 server_base.cc:1034] running on GCE node
W20240502 14:00:41.496645 3101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:41.497074 3099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:41.497246 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:00:41.497289 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:00:41.497309 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658441497309 us; error 0 us; skew 500 ppm
I20240502 14:00:41.498080 581 webserver.cc:460] Webserver started at http://127.0.145.126:33889/ using document root <none> and password file <none>
I20240502 14:00:41.498203 581 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:41.498247 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:41.498312 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:41.498651 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658376531600-581-0/minicluster-data/master-0-root/instance:
uuid: "a06a9ed26f9849b7a403284df788e156"
format_stamp: "Formatted at 2024-05-02 14:00:41 on dist-test-slave-1z5j"
I20240502 14:00:41.499917 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:41.500674 3106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:41.500900 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:41.500958 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658376531600-581-0/minicluster-data/master-0-root
uuid: "a06a9ed26f9849b7a403284df788e156"
format_stamp: "Formatted at 2024-05-02 14:00:41 on dist-test-slave-1z5j"
I20240502 14:00:41.501021 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658376531600-581-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658376531600-581-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658376531600-581-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:41.508832 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:41.509102 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:41.513979 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.126:36291
I20240502 14:00:41.515219 3158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.126:36291 every 8 connection(s)
I20240502 14:00:41.515612 3159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:41.517270 3159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a06a9ed26f9849b7a403284df788e156: Bootstrap starting.
I20240502 14:00:41.517931 3159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a06a9ed26f9849b7a403284df788e156: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:41.518761 3159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a06a9ed26f9849b7a403284df788e156: No bootstrap required, opened a new log
I20240502 14:00:41.519085 3159 raft_consensus.cc:348] T 00000000000000000000000000000000 P a06a9ed26f9849b7a403284df788e156 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a06a9ed26f9849b7a403284df788e156" member_type: VOTER }
I20240502 14:00:41.519161 3159 raft_consensus.cc:374] T 00000000000000000000000000000000 P a06a9ed26f9849b7a403284df788e156 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:41.519182 3159 raft_consensus.cc:729] T 00000000000000000000000000000000 P a06a9ed26f9849b7a403284df788e156 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a06a9ed26f9849b7a403284df788e156, State: Initialized, Role: FOLLOWER
I20240502 14:00:41.519304 3159 consensus_queue.cc:260] T 00000000000000000000000000000000 P a06a9ed26f9849b7a403284df788e156 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a06a9ed26f9849b7a403284df788e156" member_type: VOTER }
I20240502 14:00:41.519377 3159 raft_consensus.cc:388] T 00000000000000000000000000000000 P a06a9ed26f9849b7a403284df788e156 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:00:41.519412 3159 raft_consensus.cc:482] T 00000000000000000000000000000000 P a06a9ed26f9849b7a403284df788e156 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:00:41.519459 3159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a06a9ed26f9849b7a403284df788e156 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:41.520017 3159 raft_consensus.cc:504] T 00000000000000000000000000000000 P a06a9ed26f9849b7a403284df788e156 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a06a9ed26f9849b7a403284df788e156" member_type: VOTER }
I20240502 14:00:41.520124 3159 leader_election.cc:302] T 00000000000000000000000000000000 P a06a9ed26f9849b7a403284df788e156 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a06a9ed26f9849b7a403284df788e156; no voters:
I20240502 14:00:41.520275 3159 leader_election.cc:288] T 00000000000000000000000000000000 P a06a9ed26f9849b7a403284df788e156 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:00:41.520388 3162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a06a9ed26f9849b7a403284df788e156 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:41.520551 3162 raft_consensus.cc:686] T 00000000000000000000000000000000 P a06a9ed26f9849b7a403284df788e156 [term 1 LEADER]: Becoming Leader. State: Replica: a06a9ed26f9849b7a403284df788e156, State: Running, Role: LEADER
I20240502 14:00:41.520663 3162 consensus_queue.cc:237] T 00000000000000000000000000000000 P a06a9ed26f9849b7a403284df788e156 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a06a9ed26f9849b7a403284df788e156" member_type: VOTER }
I20240502 14:00:41.520848 3159 sys_catalog.cc:564] T 00000000000000000000000000000000 P a06a9ed26f9849b7a403284df788e156 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:00:41.521059 3163 sys_catalog.cc:455] T 00000000000000000000000000000000 P a06a9ed26f9849b7a403284df788e156 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a06a9ed26f9849b7a403284df788e156" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a06a9ed26f9849b7a403284df788e156" member_type: VOTER } }
I20240502 14:00:41.521162 3163 sys_catalog.cc:458] T 00000000000000000000000000000000 P a06a9ed26f9849b7a403284df788e156 [sys.catalog]: This master's current role is: LEADER
I20240502 14:00:41.521073 3164 sys_catalog.cc:455] T 00000000000000000000000000000000 P a06a9ed26f9849b7a403284df788e156 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a06a9ed26f9849b7a403284df788e156. Latest consensus state: current_term: 1 leader_uuid: "a06a9ed26f9849b7a403284df788e156" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a06a9ed26f9849b7a403284df788e156" member_type: VOTER } }
I20240502 14:00:41.521215 3164 sys_catalog.cc:458] T 00000000000000000000000000000000 P a06a9ed26f9849b7a403284df788e156 [sys.catalog]: This master's current role is: LEADER
I20240502 14:00:41.521405 3166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:00:41.522132 3166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:00:41.522548 581 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 14:00:41.523761 3166 catalog_manager.cc:1347] Generated new cluster ID: a54e1de73c8144aaaf09e3405fa0b42f
I20240502 14:00:41.523813 3166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:00:41.547633 3166 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:00:41.548107 3166 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:00:41.552187 3166 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a06a9ed26f9849b7a403284df788e156: Generated new TSK 0
I20240502 14:00:41.552315 3166 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:00:41.554512 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:41.556526 3180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:41.556936 3183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:41.557219 3181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:41.557680 581 server_base.cc:1034] running on GCE node
I20240502 14:00:41.557814 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:00:41.557849 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:00:41.557868 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658441557868 us; error 0 us; skew 500 ppm
I20240502 14:00:41.558669 581 webserver.cc:460] Webserver started at http://127.0.145.65:37021/ using document root <none> and password file <none>
I20240502 14:00:41.558795 581 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:41.558837 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:41.558907 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:41.559226 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658376531600-581-0/minicluster-data/ts-0-root/instance:
uuid: "dd87fb096c364ff5bb88796cfce8bc29"
format_stamp: "Formatted at 2024-05-02 14:00:41 on dist-test-slave-1z5j"
I20240502 14:00:41.560432 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:41.561225 3188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:41.561436 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:41.561494 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658376531600-581-0/minicluster-data/ts-0-root
uuid: "dd87fb096c364ff5bb88796cfce8bc29"
format_stamp: "Formatted at 2024-05-02 14:00:41 on dist-test-slave-1z5j"
I20240502 14:00:41.561551 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658376531600-581-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658376531600-581-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658376531600-581-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:41.570255 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:41.570503 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:41.570719 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:41.571084 581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:41.571118 581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:41.571152 581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:41.571185 581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:41.574723 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.65:40917
I20240502 14:00:41.574744 3253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.65:40917 every 8 connection(s)
I20240502 14:00:41.575363 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:41.576915 3258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:41.577301 3261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:41.577737 581 server_base.cc:1034] running on GCE node
W20240502 14:00:41.578090 3259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:41.578281 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:00:41.578325 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:00:41.578343 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658441578343 us; error 0 us; skew 500 ppm
I20240502 14:00:41.579211 581 webserver.cc:460] Webserver started at http://127.0.145.66:42479/ using document root <none> and password file <none>
I20240502 14:00:41.579340 581 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:41.579391 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:41.579421 3254 heartbeater.cc:347] Connected to a master server at 127.0.145.126:36291
I20240502 14:00:41.579445 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:41.579519 3254 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:41.579687 3254 heartbeater.cc:511] Master 127.0.145.126:36291 requested a full tablet report, sending...
I20240502 14:00:41.579759 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658376531600-581-0/minicluster-data/ts-1-root/instance:
uuid: "44064ffb73774cbf9c687a69ee89d6ab"
format_stamp: "Formatted at 2024-05-02 14:00:41 on dist-test-slave-1z5j"
I20240502 14:00:41.580257 3123 ts_manager.cc:194] Registered new tserver with Master: dd87fb096c364ff5bb88796cfce8bc29 (127.0.145.65:40917)
I20240502 14:00:41.580929 3123 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36534
I20240502 14:00:41.581017 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:41.581869 3266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:41.582074 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:41.582134 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658376531600-581-0/minicluster-data/ts-1-root
uuid: "44064ffb73774cbf9c687a69ee89d6ab"
format_stamp: "Formatted at 2024-05-02 14:00:41 on dist-test-slave-1z5j"
I20240502 14:00:41.582190 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658376531600-581-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658376531600-581-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658376531600-581-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:41.585510 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:41.585754 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:41.585983 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:41.586333 581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:41.586367 581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:41.586401 581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:41.586436 581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:41.590333 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.66:41021
I20240502 14:00:41.590397 3329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.66:41021 every 8 connection(s)
I20240502 14:00:41.590998 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:41.592882 3334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:41.593130 3333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:41.593545 3336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:41.594208 581 server_base.cc:1034] running on GCE node
I20240502 14:00:41.594377 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:00:41.594434 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:00:41.594453 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658441594453 us; error 0 us; skew 500 ppm
I20240502 14:00:41.594720 3330 heartbeater.cc:347] Connected to a master server at 127.0.145.126:36291
I20240502 14:00:41.594789 3330 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:41.594967 3330 heartbeater.cc:511] Master 127.0.145.126:36291 requested a full tablet report, sending...
I20240502 14:00:41.595273 581 webserver.cc:460] Webserver started at http://127.0.145.67:34649/ using document root <none> and password file <none>
I20240502 14:00:41.595393 581 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:41.595443 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:41.595511 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:41.595548 3123 ts_manager.cc:194] Registered new tserver with Master: 44064ffb73774cbf9c687a69ee89d6ab (127.0.145.66:41021)
I20240502 14:00:41.595902 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658376531600-581-0/minicluster-data/ts-2-root/instance:
uuid: "532ef218f8ed4788b7856017fa673b51"
format_stamp: "Formatted at 2024-05-02 14:00:41 on dist-test-slave-1z5j"
I20240502 14:00:41.596159 3123 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36548
I20240502 14:00:41.597267 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:41.598028 3341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:41.598253 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:41.598310 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658376531600-581-0/minicluster-data/ts-2-root
uuid: "532ef218f8ed4788b7856017fa673b51"
format_stamp: "Formatted at 2024-05-02 14:00:41 on dist-test-slave-1z5j"
I20240502 14:00:41.598371 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658376531600-581-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658376531600-581-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658376531600-581-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:41.605691 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:41.605979 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:41.606220 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:41.606595 581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:41.606631 581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:41.606666 581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:41.606698 581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:41.610940 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.67:38839
I20240502 14:00:41.610980 3405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.67:38839 every 8 connection(s)
I20240502 14:00:41.611671 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:41.613497 3409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:41.614013 3410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:41.614660 581 server_base.cc:1034] running on GCE node
W20240502 14:00:41.614948 3412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:41.615150 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:00:41.615207 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:00:41.615229 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658441615229 us; error 0 us; skew 500 ppm
I20240502 14:00:41.615271 3406 heartbeater.cc:347] Connected to a master server at 127.0.145.126:36291
I20240502 14:00:41.615360 3406 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:41.615556 3406 heartbeater.cc:511] Master 127.0.145.126:36291 requested a full tablet report, sending...
I20240502 14:00:41.616113 3123 ts_manager.cc:194] Registered new tserver with Master: 532ef218f8ed4788b7856017fa673b51 (127.0.145.67:38839)
I20240502 14:00:41.616114 581 webserver.cc:460] Webserver started at http://127.0.145.68:44101/ using document root <none> and password file <none>
I20240502 14:00:41.616279 581 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:41.616322 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:41.616389 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:41.616645 3123 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36562
I20240502 14:00:41.616735 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658376531600-581-0/minicluster-data/ts-3-root/instance:
uuid: "2bd5849661484595b55828a77f1bdb01"
format_stamp: "Formatted at 2024-05-02 14:00:41 on dist-test-slave-1z5j"
I20240502 14:00:41.618002 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:41.618758 3417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:41.618973 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:41.619032 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658376531600-581-0/minicluster-data/ts-3-root
uuid: "2bd5849661484595b55828a77f1bdb01"
format_stamp: "Formatted at 2024-05-02 14:00:41 on dist-test-slave-1z5j"
I20240502 14:00:41.619091 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658376531600-581-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658376531600-581-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658376531600-581-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:41.627308 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:41.627595 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:41.627830 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:41.628207 581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:41.628243 581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:41.628278 581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:41.628312 581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:41.632472 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.68:38861
I20240502 14:00:41.632512 3480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.68:38861 every 8 connection(s)
I20240502 14:00:41.636248 3481 heartbeater.cc:347] Connected to a master server at 127.0.145.126:36291
I20240502 14:00:41.636322 3481 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:41.636499 3481 heartbeater.cc:511] Master 127.0.145.126:36291 requested a full tablet report, sending...
I20240502 14:00:41.637033 3123 ts_manager.cc:194] Registered new tserver with Master: 2bd5849661484595b55828a77f1bdb01 (127.0.145.68:38861)
I20240502 14:00:41.637393 581 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.004545277s
I20240502 14:00:41.637539 3123 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36574
I20240502 14:00:42.581687 3254 heartbeater.cc:503] Master 127.0.145.126:36291 was elected leader, sending a full tablet report...
I20240502 14:00:42.596942 3330 heartbeater.cc:503] Master 127.0.145.126:36291 was elected leader, sending a full tablet report...
I20240502 14:00:42.617404 3406 heartbeater.cc:503] Master 127.0.145.126:36291 was elected leader, sending a full tablet report...
I20240502 14:00:42.638290 3481 heartbeater.cc:503] Master 127.0.145.126:36291 was elected leader, sending a full tablet report...
I20240502 14:00:42.666880 581 test_util.cc:268] Using random seed: 123921279
I20240502 14:00:42.673446 3123 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36592:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 14:00:42.682567 3445 tablet_service.cc:1467] Processing CreateTablet for tablet fa914a31242a42f49b90a7224c6dbae9 (DEFAULT_TABLE table=test-workload [id=5584b1b2980847cd955fa12876e30e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:42.682899 3445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa914a31242a42f49b90a7224c6dbae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:42.684880 3502 tablet_bootstrap.cc:492] T fa914a31242a42f49b90a7224c6dbae9 P 2bd5849661484595b55828a77f1bdb01: Bootstrap starting.
I20240502 14:00:42.685034 3370 tablet_service.cc:1467] Processing CreateTablet for tablet fa914a31242a42f49b90a7224c6dbae9 (DEFAULT_TABLE table=test-workload [id=5584b1b2980847cd955fa12876e30e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:42.685257 3370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa914a31242a42f49b90a7224c6dbae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:42.685663 3502 tablet_bootstrap.cc:654] T fa914a31242a42f49b90a7224c6dbae9 P 2bd5849661484595b55828a77f1bdb01: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:42.686601 3502 tablet_bootstrap.cc:492] T fa914a31242a42f49b90a7224c6dbae9 P 2bd5849661484595b55828a77f1bdb01: No bootstrap required, opened a new log
I20240502 14:00:42.686681 3502 ts_tablet_manager.cc:1367] T fa914a31242a42f49b90a7224c6dbae9 P 2bd5849661484595b55828a77f1bdb01: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:42.687150 3502 raft_consensus.cc:348] T fa914a31242a42f49b90a7224c6dbae9 P 2bd5849661484595b55828a77f1bdb01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd5849661484595b55828a77f1bdb01" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 38861 } } peers { permanent_uuid: "44064ffb73774cbf9c687a69ee89d6ab" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41021 } } peers { permanent_uuid: "532ef218f8ed4788b7856017fa673b51" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 38839 } }
I20240502 14:00:42.687268 3502 raft_consensus.cc:374] T fa914a31242a42f49b90a7224c6dbae9 P 2bd5849661484595b55828a77f1bdb01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:42.687312 3502 raft_consensus.cc:729] T fa914a31242a42f49b90a7224c6dbae9 P 2bd5849661484595b55828a77f1bdb01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bd5849661484595b55828a77f1bdb01, State: Initialized, Role: FOLLOWER
I20240502 14:00:42.687417 3502 consensus_queue.cc:260] T fa914a31242a42f49b90a7224c6dbae9 P 2bd5849661484595b55828a77f1bdb01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd5849661484595b55828a77f1bdb01" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 38861 } } peers { permanent_uuid: "44064ffb73774cbf9c687a69ee89d6ab" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41021 } } peers { permanent_uuid: "532ef218f8ed4788b7856017fa673b51" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 38839 } }
I20240502 14:00:42.687430 3294 tablet_service.cc:1467] Processing CreateTablet for tablet fa914a31242a42f49b90a7224c6dbae9 (DEFAULT_TABLE table=test-workload [id=5584b1b2980847cd955fa12876e30e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:42.687682 3294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa914a31242a42f49b90a7224c6dbae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:42.687754 3502 ts_tablet_manager.cc:1398] T fa914a31242a42f49b90a7224c6dbae9 P 2bd5849661484595b55828a77f1bdb01: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:42.687976 3504 tablet_bootstrap.cc:492] T fa914a31242a42f49b90a7224c6dbae9 P 532ef218f8ed4788b7856017fa673b51: Bootstrap starting.
I20240502 14:00:42.688884 3504 tablet_bootstrap.cc:654] T fa914a31242a42f49b90a7224c6dbae9 P 532ef218f8ed4788b7856017fa673b51: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:42.689528 3506 tablet_bootstrap.cc:492] T fa914a31242a42f49b90a7224c6dbae9 P 44064ffb73774cbf9c687a69ee89d6ab: Bootstrap starting.
I20240502 14:00:42.689910 3504 tablet_bootstrap.cc:492] T fa914a31242a42f49b90a7224c6dbae9 P 532ef218f8ed4788b7856017fa673b51: No bootstrap required, opened a new log
I20240502 14:00:42.690014 3504 ts_tablet_manager.cc:1367] T fa914a31242a42f49b90a7224c6dbae9 P 532ef218f8ed4788b7856017fa673b51: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:42.690326 3506 tablet_bootstrap.cc:654] T fa914a31242a42f49b90a7224c6dbae9 P 44064ffb73774cbf9c687a69ee89d6ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:42.690359 3504 raft_consensus.cc:348] T fa914a31242a42f49b90a7224c6dbae9 P 532ef218f8ed4788b7856017fa673b51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd5849661484595b55828a77f1bdb01" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 38861 } } peers { permanent_uuid: "44064ffb73774cbf9c687a69ee89d6ab" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41021 } } peers { permanent_uuid: "532ef218f8ed4788b7856017fa673b51" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 38839 } }
I20240502 14:00:42.690444 3504 raft_consensus.cc:374] T fa914a31242a42f49b90a7224c6dbae9 P 532ef218f8ed4788b7856017fa673b51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:42.690464 3504 raft_consensus.cc:729] T fa914a31242a42f49b90a7224c6dbae9 P 532ef218f8ed4788b7856017fa673b51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 532ef218f8ed4788b7856017fa673b51, State: Initialized, Role: FOLLOWER
I20240502 14:00:42.690570 3504 consensus_queue.cc:260] T fa914a31242a42f49b90a7224c6dbae9 P 532ef218f8ed4788b7856017fa673b51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd5849661484595b55828a77f1bdb01" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 38861 } } peers { permanent_uuid: "44064ffb73774cbf9c687a69ee89d6ab" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41021 } } peers { permanent_uuid: "532ef218f8ed4788b7856017fa673b51" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 38839 } }
I20240502 14:00:42.690874 3504 ts_tablet_manager.cc:1398] T fa914a31242a42f49b90a7224c6dbae9 P 532ef218f8ed4788b7856017fa673b51: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:42.691746 3506 tablet_bootstrap.cc:492] T fa914a31242a42f49b90a7224c6dbae9 P 44064ffb73774cbf9c687a69ee89d6ab: No bootstrap required, opened a new log
I20240502 14:00:42.691841 3506 ts_tablet_manager.cc:1367] T fa914a31242a42f49b90a7224c6dbae9 P 44064ffb73774cbf9c687a69ee89d6ab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:42.692472 3506 raft_consensus.cc:348] T fa914a31242a42f49b90a7224c6dbae9 P 44064ffb73774cbf9c687a69ee89d6ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd5849661484595b55828a77f1bdb01" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 38861 } } peers { permanent_uuid: "44064ffb73774cbf9c687a69ee89d6ab" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41021 } } peers { permanent_uuid: "532ef218f8ed4788b7856017fa673b51" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 38839 } }
I20240502 14:00:42.692579 3506 raft_consensus.cc:374] T fa914a31242a42f49b90a7224c6dbae9 P 44064ffb73774cbf9c687a69ee89d6ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:42.692623 3506 raft_consensus.cc:729] T fa914a31242a42f49b90a7224c6dbae9 P 44064ffb73774cbf9c687a69ee89d6ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44064ffb73774cbf9c687a69ee89d6ab, State: Initialized, Role: FOLLOWER
I20240502 14:00:42.692739 3506 consensus_queue.cc:260] T fa914a31242a42f49b90a7224c6dbae9 P 44064ffb73774cbf9c687a69ee89d6ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd5849661484595b55828a77f1bdb01" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 38861 } } peers { permanent_uuid: "44064ffb73774cbf9c687a69ee89d6ab" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41021 } } peers { permanent_uuid: "532ef218f8ed4788b7856017fa673b51" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 38839 } }
I20240502 14:00:42.693172 3506 ts_tablet_manager.cc:1398] T fa914a31242a42f49b90a7224c6dbae9 P 44064ffb73774cbf9c687a69ee89d6ab: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240502 14:00:42.841408 3331 tablet.cc:2369] T fa914a31242a42f49b90a7224c6dbae9 P 44064ffb73774cbf9c687a69ee89d6ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:00:42.845752 3505 raft_consensus.cc:482] T fa914a31242a42f49b90a7224c6dbae9 P 2bd5849661484595b55828a77f1bdb01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:42.845854 3505 raft_consensus.cc:504] T fa914a31242a42f49b90a7224c6dbae9 P 2bd5849661484595b55828a77f1bdb01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd5849661484595b55828a77f1bdb01" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 38861 } } peers { permanent_uuid: "44064ffb73774cbf9c687a69ee89d6ab" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41021 } } peers { permanent_uuid: "532ef218f8ed4788b7856017fa673b51" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 38839 } }
I20240502 14:00:42.846273 3505 leader_election.cc:288] T fa914a31242a42f49b90a7224c6dbae9 P 2bd5849661484595b55828a77f1bdb01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44064ffb73774cbf9c687a69ee89d6ab (127.0.145.66:41021), 532ef218f8ed4788b7856017fa673b51 (127.0.145.67:38839)
I20240502 14:00:42.849470 3304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa914a31242a42f49b90a7224c6dbae9" candidate_uuid: "2bd5849661484595b55828a77f1bdb01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44064ffb73774cbf9c687a69ee89d6ab" is_pre_election: true
I20240502 14:00:42.849470 3380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa914a31242a42f49b90a7224c6dbae9" candidate_uuid: "2bd5849661484595b55828a77f1bdb01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "532ef218f8ed4788b7856017fa673b51" is_pre_election: true
I20240502 14:00:42.849642 3304 raft_consensus.cc:2454] T fa914a31242a42f49b90a7224c6dbae9 P 44064ffb73774cbf9c687a69ee89d6ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bd5849661484595b55828a77f1bdb01 in term 0.
I20240502 14:00:42.849649 3380 raft_consensus.cc:2454] T fa914a31242a42f49b90a7224c6dbae9 P 532ef218f8ed4788b7856017fa673b51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bd5849661484595b55828a77f1bdb01 in term 0.
I20240502 14:00:42.849947 3421 leader_election.cc:302] T fa914a31242a42f49b90a7224c6dbae9 P 2bd5849661484595b55828a77f1bdb01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bd5849661484595b55828a77f1bdb01, 44064ffb73774cbf9c687a69ee89d6ab; no voters:
I20240502 14:00:42.850124 3505 raft_consensus.cc:2781] T fa914a31242a42f49b90a7224c6dbae9 P 2bd5849661484595b55828a77f1bdb01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:42.850186 3505 raft_consensus.cc:482] T fa914a31242a42f49b90a7224c6dbae9 P 2bd5849661484595b55828a77f1bdb01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:42.850224 3505 raft_consensus.cc:3037] T fa914a31242a42f49b90a7224c6dbae9 P 2bd5849661484595b55828a77f1bdb01 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:42.850980 3505 raft_consensus.cc:504] T fa914a31242a42f49b90a7224c6dbae9 P 2bd5849661484595b55828a77f1bdb01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd5849661484595b55828a77f1bdb01" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 38861 } } peers { permanent_uuid: "44064ffb73774cbf9c687a69ee89d6ab" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41021 } } peers { permanent_uuid: "532ef218f8ed4788b7856017fa673b51" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 38839 } }
I20240502 14:00:42.851274 3505 leader_election.cc:288] T fa914a31242a42f49b90a7224c6dbae9 P 2bd5849661484595b55828a77f1bdb01 [CANDIDATE]: Term 1 election: Requested vote from peers 44064ffb73774cbf9c687a69ee89d6ab (127.0.145.66:41021), 532ef218f8ed4788b7856017fa673b51 (127.0.145.67:38839)
I20240502 14:00:42.851554 3380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa914a31242a42f49b90a7224c6dbae9" candidate_uuid: "2bd5849661484595b55828a77f1bdb01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "532ef218f8ed4788b7856017fa673b51"
I20240502 14:00:42.851554 3304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa914a31242a42f49b90a7224c6dbae9" candidate_uuid: "2bd5849661484595b55828a77f1bdb01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44064ffb73774cbf9c687a69ee89d6ab"
I20240502 14:00:42.851688 3380 raft_consensus.cc:3037] T fa914a31242a42f49b90a7224c6dbae9 P 532ef218f8ed4788b7856017fa673b51 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:42.851689 3304 raft_consensus.cc:3037] T fa914a31242a42f49b90a7224c6dbae9 P 44064ffb73774cbf9c687a69ee89d6ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:42.852595 3304 raft_consensus.cc:2454] T fa914a31242a42f49b90a7224c6dbae9 P 44064ffb73774cbf9c687a69ee89d6ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bd5849661484595b55828a77f1bdb01 in term 1.
I20240502 14:00:42.852595 3380 raft_consensus.cc:2454] T fa914a31242a42f49b90a7224c6dbae9 P 532ef218f8ed4788b7856017fa673b51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bd5849661484595b55828a77f1bdb01 in term 1.
I20240502 14:00:42.852874 3421 leader_election.cc:302] T fa914a31242a42f49b90a7224c6dbae9 P 2bd5849661484595b55828a77f1bdb01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bd5849661484595b55828a77f1bdb01, 44064ffb73774cbf9c687a69ee89d6ab; no voters:
I20240502 14:00:42.853034 3505 raft_consensus.cc:2781] T fa914a31242a42f49b90a7224c6dbae9 P 2bd5849661484595b55828a77f1bdb01 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:42.853209 3505 raft_consensus.cc:686] T fa914a31242a42f49b90a7224c6dbae9 P 2bd5849661484595b55828a77f1bdb01 [term 1 LEADER]: Becoming Leader. State: Replica: 2bd5849661484595b55828a77f1bdb01, State: Running, Role: LEADER
I20240502 14:00:42.853338 3505 consensus_queue.cc:237] T fa914a31242a42f49b90a7224c6dbae9 P 2bd5849661484595b55828a77f1bdb01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd5849661484595b55828a77f1bdb01" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 38861 } } peers { permanent_uuid: "44064ffb73774cbf9c687a69ee89d6ab" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41021 } } peers { permanent_uuid: "532ef218f8ed4788b7856017fa673b51" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 38839 } }
I20240502 14:00:42.854655 3123 catalog_manager.cc:5484] T fa914a31242a42f49b90a7224c6dbae9 P 2bd5849661484595b55828a77f1bdb01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2bd5849661484595b55828a77f1bdb01 (127.0.145.68). New cstate: current_term: 1 leader_uuid: "2bd5849661484595b55828a77f1bdb01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd5849661484595b55828a77f1bdb01" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 38861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44064ffb73774cbf9c687a69ee89d6ab" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "532ef218f8ed4788b7856017fa673b51" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 38839 } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:43.312994 3505 consensus_queue.cc:1035] T fa914a31242a42f49b90a7224c6dbae9 P 2bd5849661484595b55828a77f1bdb01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44064ffb73774cbf9c687a69ee89d6ab" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:43.315932 3519 consensus_queue.cc:1035] T fa914a31242a42f49b90a7224c6dbae9 P 2bd5849661484595b55828a77f1bdb01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "532ef218f8ed4788b7856017fa673b51" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 38839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:43.522642 3177 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:00:43.522927 3177 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:00:43.523003 3177 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:00:45.523283 3177 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:00:45.523600 3177 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:00:45.523666 3177 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:00:47.523994 3177 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:00:47.524298 3177 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:00:47.524366 3177 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:00:49.524644 3177 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:00:49.524964 3177 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:00:49.525031 3177 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:00:51.525295 3177 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:00:51.525599 3177 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:00:51.525669 3177 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:00:53.525923 3177 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:00:53.526223 3177 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:00:53.526291 3177 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:00:53.911634 581 tablet_server.cc:178] TabletServer@127.0.145.65:0 shutting down...
I20240502 14:00:53.914866 581 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:00:53.926429 581 tablet_server.cc:194] TabletServer@127.0.145.65:0 shutdown complete.
I20240502 14:00:53.928009 581 tablet_server.cc:178] TabletServer@127.0.145.66:0 shutting down...
I20240502 14:00:53.930935 581 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:00:53.931087 581 tablet_replica.cc:330] T fa914a31242a42f49b90a7224c6dbae9 P 44064ffb73774cbf9c687a69ee89d6ab: stopping tablet replica
I20240502 14:00:53.931178 581 raft_consensus.cc:2229] T fa914a31242a42f49b90a7224c6dbae9 P 44064ffb73774cbf9c687a69ee89d6ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:53.931272 581 raft_consensus.cc:2258] T fa914a31242a42f49b90a7224c6dbae9 P 44064ffb73774cbf9c687a69ee89d6ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:53.943131 581 tablet_server.cc:194] TabletServer@127.0.145.66:0 shutdown complete.
I20240502 14:00:53.944612 581 tablet_server.cc:178] TabletServer@127.0.145.67:0 shutting down...
I20240502 14:00:53.947572 581 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:00:53.947716 581 tablet_replica.cc:330] T fa914a31242a42f49b90a7224c6dbae9 P 532ef218f8ed4788b7856017fa673b51: stopping tablet replica
W20240502 14:00:53.947719 3420 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 9 similar messages]
I20240502 14:00:53.947844 581 raft_consensus.cc:2229] T fa914a31242a42f49b90a7224c6dbae9 P 532ef218f8ed4788b7856017fa673b51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:53.947953 581 raft_consensus.cc:2258] T fa914a31242a42f49b90a7224c6dbae9 P 532ef218f8ed4788b7856017fa673b51 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 14:00:53.948393 3420 consensus_peers.cc:485] T fa914a31242a42f49b90a7224c6dbae9 P 2bd5849661484595b55828a77f1bdb01 -> Peer 532ef218f8ed4788b7856017fa673b51 (127.0.145.67:38839): Couldn't send request to peer 532ef218f8ed4788b7856017fa673b51. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:53.959793 581 tablet_server.cc:194] TabletServer@127.0.145.67:0 shutdown complete.
I20240502 14:00:53.961305 581 tablet_server.cc:178] TabletServer@127.0.145.68:0 shutting down...
I20240502 14:00:53.964339 581 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:00:53.964478 581 tablet_replica.cc:330] T fa914a31242a42f49b90a7224c6dbae9 P 2bd5849661484595b55828a77f1bdb01: stopping tablet replica
I20240502 14:00:53.964563 581 raft_consensus.cc:2229] T fa914a31242a42f49b90a7224c6dbae9 P 2bd5849661484595b55828a77f1bdb01 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:53.964715 581 raft_consensus.cc:2258] T fa914a31242a42f49b90a7224c6dbae9 P 2bd5849661484595b55828a77f1bdb01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:53.976523 581 tablet_server.cc:194] TabletServer@127.0.145.68:0 shutdown complete.
I20240502 14:00:53.978053 581 master.cc:537] Master@127.0.145.126:36291 shutting down...
I20240502 14:00:53.980726 581 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a06a9ed26f9849b7a403284df788e156 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:53.980849 581 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a06a9ed26f9849b7a403284df788e156 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:53.980897 581 tablet_replica.cc:330] T 00000000000000000000000000000000 P a06a9ed26f9849b7a403284df788e156: stopping tablet replica
I20240502 14:00:53.992570 581 master.cc:552] Master@127.0.145.126:36291 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12504 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20240502 14:00:53.997772 581 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.145.126:34985
I20240502 14:00:53.998050 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:53.999812 3531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:54.000355 3533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:54.000806 3530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:54.001217 581 server_base.cc:1034] running on GCE node
I20240502 14:00:54.001338 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:00:54.001369 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:00:54.001382 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658454001382 us; error 0 us; skew 500 ppm
I20240502 14:00:54.002140 581 webserver.cc:460] Webserver started at http://127.0.145.126:34381/ using document root <none> and password file <none>
I20240502 14:00:54.002260 581 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:54.002305 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:54.002372 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:54.002693 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658376531600-581-0/minicluster-data/master-0-root/instance:
uuid: "3aa40935fdaa4d0dbd4066fc61abb556"
format_stamp: "Formatted at 2024-05-02 14:00:53 on dist-test-slave-1z5j"
I20240502 14:00:54.003906 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:54.004660 3538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:54.004913 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:54.004982 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658376531600-581-0/minicluster-data/master-0-root
uuid: "3aa40935fdaa4d0dbd4066fc61abb556"
format_stamp: "Formatted at 2024-05-02 14:00:53 on dist-test-slave-1z5j"
I20240502 14:00:54.005044 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658376531600-581-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658376531600-581-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658376531600-581-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:54.023844 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:54.024092 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:54.027539 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.126:34985
I20240502 14:00:54.029196 3590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.126:34985 every 8 connection(s)
I20240502 14:00:54.029574 3591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:54.031212 3591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aa40935fdaa4d0dbd4066fc61abb556: Bootstrap starting.
I20240502 14:00:54.031855 3591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3aa40935fdaa4d0dbd4066fc61abb556: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:54.032670 3591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aa40935fdaa4d0dbd4066fc61abb556: No bootstrap required, opened a new log
I20240502 14:00:54.033016 3591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3aa40935fdaa4d0dbd4066fc61abb556 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa40935fdaa4d0dbd4066fc61abb556" member_type: VOTER }
I20240502 14:00:54.033094 3591 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3aa40935fdaa4d0dbd4066fc61abb556 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:54.033124 3591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3aa40935fdaa4d0dbd4066fc61abb556 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aa40935fdaa4d0dbd4066fc61abb556, State: Initialized, Role: FOLLOWER
I20240502 14:00:54.033241 3591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3aa40935fdaa4d0dbd4066fc61abb556 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa40935fdaa4d0dbd4066fc61abb556" member_type: VOTER }
I20240502 14:00:54.033301 3591 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3aa40935fdaa4d0dbd4066fc61abb556 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:00:54.033334 3591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3aa40935fdaa4d0dbd4066fc61abb556 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:00:54.033380 3591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3aa40935fdaa4d0dbd4066fc61abb556 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:54.033955 3591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3aa40935fdaa4d0dbd4066fc61abb556 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa40935fdaa4d0dbd4066fc61abb556" member_type: VOTER }
I20240502 14:00:54.034061 3591 leader_election.cc:302] T 00000000000000000000000000000000 P 3aa40935fdaa4d0dbd4066fc61abb556 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3aa40935fdaa4d0dbd4066fc61abb556; no voters:
I20240502 14:00:54.034214 3591 leader_election.cc:288] T 00000000000000000000000000000000 P 3aa40935fdaa4d0dbd4066fc61abb556 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:00:54.034298 3594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3aa40935fdaa4d0dbd4066fc61abb556 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:54.034461 3594 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3aa40935fdaa4d0dbd4066fc61abb556 [term 1 LEADER]: Becoming Leader. State: Replica: 3aa40935fdaa4d0dbd4066fc61abb556, State: Running, Role: LEADER
I20240502 14:00:54.034577 3591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3aa40935fdaa4d0dbd4066fc61abb556 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:00:54.034569 3594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3aa40935fdaa4d0dbd4066fc61abb556 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa40935fdaa4d0dbd4066fc61abb556" member_type: VOTER }
I20240502 14:00:54.035075 3595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aa40935fdaa4d0dbd4066fc61abb556 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3aa40935fdaa4d0dbd4066fc61abb556" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa40935fdaa4d0dbd4066fc61abb556" member_type: VOTER } }
I20240502 14:00:54.035149 3595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aa40935fdaa4d0dbd4066fc61abb556 [sys.catalog]: This master's current role is: LEADER
I20240502 14:00:54.035318 3596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aa40935fdaa4d0dbd4066fc61abb556 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3aa40935fdaa4d0dbd4066fc61abb556. Latest consensus state: current_term: 1 leader_uuid: "3aa40935fdaa4d0dbd4066fc61abb556" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa40935fdaa4d0dbd4066fc61abb556" member_type: VOTER } }
I20240502 14:00:54.035393 3596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aa40935fdaa4d0dbd4066fc61abb556 [sys.catalog]: This master's current role is: LEADER
I20240502 14:00:54.035694 3599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:00:54.036432 3599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:00:54.036623 581 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 14:00:54.038149 3599 catalog_manager.cc:1347] Generated new cluster ID: a706eaef151149efa28757d509b5143b
I20240502 14:00:54.038194 3599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:00:54.053992 3599 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:00:54.054455 3599 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:00:54.066414 3599 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3aa40935fdaa4d0dbd4066fc61abb556: Generated new TSK 0
I20240502 14:00:54.066542 3599 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:00:54.068562 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:54.070633 3613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:54.070937 3612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:54.071390 3615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:54.071808 581 server_base.cc:1034] running on GCE node
I20240502 14:00:54.071941 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:00:54.071981 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:00:54.071996 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658454071996 us; error 0 us; skew 500 ppm
I20240502 14:00:54.072831 581 webserver.cc:460] Webserver started at http://127.0.145.65:44119/ using document root <none> and password file <none>
I20240502 14:00:54.072962 581 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:54.073004 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:54.073073 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:54.073379 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658376531600-581-0/minicluster-data/ts-0-root/instance:
uuid: "9cd2dca60ac5419f85ac80cb4d700351"
format_stamp: "Formatted at 2024-05-02 14:00:54 on dist-test-slave-1z5j"
I20240502 14:00:54.074602 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 14:00:54.075369 3620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:54.075623 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:54.075690 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658376531600-581-0/minicluster-data/ts-0-root
uuid: "9cd2dca60ac5419f85ac80cb4d700351"
format_stamp: "Formatted at 2024-05-02 14:00:54 on dist-test-slave-1z5j"
I20240502 14:00:54.075742 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658376531600-581-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658376531600-581-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658376531600-581-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:54.090400 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:54.090642 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:54.090834 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:54.091171 581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:54.091203 581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:54.091229 581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:54.091248 581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:54.094885 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.65:34781
I20240502 14:00:54.094928 3683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.65:34781 every 8 connection(s)
I20240502 14:00:54.095558 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:54.097250 3688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:54.097555 3691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:54.097942 3689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:54.098382 581 server_base.cc:1034] running on GCE node
I20240502 14:00:54.098537 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:00:54.098584 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:00:54.098603 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658454098603 us; error 0 us; skew 500 ppm
I20240502 14:00:54.099426 581 webserver.cc:460] Webserver started at http://127.0.145.66:39095/ using document root <none> and password file <none>
I20240502 14:00:54.099566 581 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:54.099622 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:54.099685 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:54.099804 3684 heartbeater.cc:347] Connected to a master server at 127.0.145.126:34985
I20240502 14:00:54.099890 3684 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:54.100026 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658376531600-581-0/minicluster-data/ts-1-root/instance:
uuid: "d3c69cd2c012414f8d19e14545d2ebb9"
format_stamp: "Formatted at 2024-05-02 14:00:54 on dist-test-slave-1z5j"
I20240502 14:00:54.100067 3684 heartbeater.cc:511] Master 127.0.145.126:34985 requested a full tablet report, sending...
I20240502 14:00:54.100606 3555 ts_manager.cc:194] Registered new tserver with Master: 9cd2dca60ac5419f85ac80cb4d700351 (127.0.145.65:34781)
I20240502 14:00:54.101341 3555 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43676
I20240502 14:00:54.101383 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 14:00:54.102209 3696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:54.102393 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:54.102452 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658376531600-581-0/minicluster-data/ts-1-root
uuid: "d3c69cd2c012414f8d19e14545d2ebb9"
format_stamp: "Formatted at 2024-05-02 14:00:54 on dist-test-slave-1z5j"
I20240502 14:00:54.102514 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658376531600-581-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658376531600-581-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658376531600-581-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:54.125133 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:54.125383 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:54.125619 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:54.125980 581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:54.126015 581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:54.126050 581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:54.126082 581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:54.129660 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.66:41089
I20240502 14:00:54.129716 3759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.66:41089 every 8 connection(s)
I20240502 14:00:54.130329 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:54.132020 3763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:54.132443 3764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:54.132884 3766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:54.133350 581 server_base.cc:1034] running on GCE node
I20240502 14:00:54.133486 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:00:54.133527 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:00:54.133541 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658454133541 us; error 0 us; skew 500 ppm
I20240502 14:00:54.134408 3760 heartbeater.cc:347] Connected to a master server at 127.0.145.126:34985
I20240502 14:00:54.134502 3760 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:54.134521 581 webserver.cc:460] Webserver started at http://127.0.145.67:34367/ using document root <none> and password file <none>
I20240502 14:00:54.134646 581 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:54.134682 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:54.134713 3760 heartbeater.cc:511] Master 127.0.145.126:34985 requested a full tablet report, sending...
I20240502 14:00:54.134745 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:54.135089 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658376531600-581-0/minicluster-data/ts-2-root/instance:
uuid: "44e80272573649d7bab270010b4bddad"
format_stamp: "Formatted at 2024-05-02 14:00:54 on dist-test-slave-1z5j"
I20240502 14:00:54.135267 3555 ts_manager.cc:194] Registered new tserver with Master: d3c69cd2c012414f8d19e14545d2ebb9 (127.0.145.66:41089)
I20240502 14:00:54.135879 3555 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43690
I20240502 14:00:54.136704 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:54.137486 3771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:54.137687 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:54.137743 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658376531600-581-0/minicluster-data/ts-2-root
uuid: "44e80272573649d7bab270010b4bddad"
format_stamp: "Formatted at 2024-05-02 14:00:54 on dist-test-slave-1z5j"
I20240502 14:00:54.137801 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658376531600-581-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658376531600-581-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658376531600-581-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:54.143185 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:54.143429 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:54.143644 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:54.144007 581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:54.144043 581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:54.144078 581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:54.144105 581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:54.148434 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.67:33541
I20240502 14:00:54.148488 3834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.67:33541 every 8 connection(s)
I20240502 14:00:54.151896 3835 heartbeater.cc:347] Connected to a master server at 127.0.145.126:34985
I20240502 14:00:54.151970 3835 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:54.152132 3835 heartbeater.cc:511] Master 127.0.145.126:34985 requested a full tablet report, sending...
I20240502 14:00:54.152616 3555 ts_manager.cc:194] Registered new tserver with Master: 44e80272573649d7bab270010b4bddad (127.0.145.67:33541)
I20240502 14:00:54.153080 3555 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43692
I20240502 14:00:54.153391 581 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.00456582s
I20240502 14:00:55.102150 3684 heartbeater.cc:503] Master 127.0.145.126:34985 was elected leader, sending a full tablet report...
I20240502 14:00:55.136744 3760 heartbeater.cc:503] Master 127.0.145.126:34985 was elected leader, sending a full tablet report...
I20240502 14:00:55.153782 3835 heartbeater.cc:503] Master 127.0.145.126:34985 was elected leader, sending a full tablet report...
I20240502 14:00:55.182610 581 test_util.cc:268] Using random seed: 136437007
I20240502 14:00:55.188944 3555 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43722:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:00:55.189373 3555 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:00:55.198153 3799 tablet_service.cc:1467] Processing CreateTablet for tablet 94ff128e9466454c9919d9ae6c2d8097 (DEFAULT_TABLE table=test-workload [id=e82e491264de41e9bdc2ae8d55ba0537]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:55.198153 3724 tablet_service.cc:1467] Processing CreateTablet for tablet 94ff128e9466454c9919d9ae6c2d8097 (DEFAULT_TABLE table=test-workload [id=e82e491264de41e9bdc2ae8d55ba0537]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:55.198333 3648 tablet_service.cc:1467] Processing CreateTablet for tablet 94ff128e9466454c9919d9ae6c2d8097 (DEFAULT_TABLE table=test-workload [id=e82e491264de41e9bdc2ae8d55ba0537]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:55.198470 3799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94ff128e9466454c9919d9ae6c2d8097. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:55.198478 3724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94ff128e9466454c9919d9ae6c2d8097. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:55.198539 3648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94ff128e9466454c9919d9ae6c2d8097. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:55.200793 3858 tablet_bootstrap.cc:492] T 94ff128e9466454c9919d9ae6c2d8097 P 9cd2dca60ac5419f85ac80cb4d700351: Bootstrap starting.
I20240502 14:00:55.200863 3856 tablet_bootstrap.cc:492] T 94ff128e9466454c9919d9ae6c2d8097 P d3c69cd2c012414f8d19e14545d2ebb9: Bootstrap starting.
I20240502 14:00:55.200891 3857 tablet_bootstrap.cc:492] T 94ff128e9466454c9919d9ae6c2d8097 P 44e80272573649d7bab270010b4bddad: Bootstrap starting.
I20240502 14:00:55.201648 3856 tablet_bootstrap.cc:654] T 94ff128e9466454c9919d9ae6c2d8097 P d3c69cd2c012414f8d19e14545d2ebb9: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:55.201733 3858 tablet_bootstrap.cc:654] T 94ff128e9466454c9919d9ae6c2d8097 P 9cd2dca60ac5419f85ac80cb4d700351: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:55.201733 3857 tablet_bootstrap.cc:654] T 94ff128e9466454c9919d9ae6c2d8097 P 44e80272573649d7bab270010b4bddad: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:55.202629 3856 tablet_bootstrap.cc:492] T 94ff128e9466454c9919d9ae6c2d8097 P d3c69cd2c012414f8d19e14545d2ebb9: No bootstrap required, opened a new log
I20240502 14:00:55.202744 3856 ts_tablet_manager.cc:1367] T 94ff128e9466454c9919d9ae6c2d8097 P d3c69cd2c012414f8d19e14545d2ebb9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:55.203022 3858 tablet_bootstrap.cc:492] T 94ff128e9466454c9919d9ae6c2d8097 P 9cd2dca60ac5419f85ac80cb4d700351: No bootstrap required, opened a new log
I20240502 14:00:55.203099 3858 ts_tablet_manager.cc:1367] T 94ff128e9466454c9919d9ae6c2d8097 P 9cd2dca60ac5419f85ac80cb4d700351: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:55.203140 3857 tablet_bootstrap.cc:492] T 94ff128e9466454c9919d9ae6c2d8097 P 44e80272573649d7bab270010b4bddad: No bootstrap required, opened a new log
I20240502 14:00:55.203157 3856 raft_consensus.cc:348] T 94ff128e9466454c9919d9ae6c2d8097 P d3c69cd2c012414f8d19e14545d2ebb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e80272573649d7bab270010b4bddad" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 33541 } } peers { permanent_uuid: "d3c69cd2c012414f8d19e14545d2ebb9" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41089 } } peers { permanent_uuid: "9cd2dca60ac5419f85ac80cb4d700351" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34781 } }
I20240502 14:00:55.203266 3857 ts_tablet_manager.cc:1367] T 94ff128e9466454c9919d9ae6c2d8097 P 44e80272573649d7bab270010b4bddad: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:00:55.203279 3856 raft_consensus.cc:374] T 94ff128e9466454c9919d9ae6c2d8097 P d3c69cd2c012414f8d19e14545d2ebb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:55.203370 3856 raft_consensus.cc:729] T 94ff128e9466454c9919d9ae6c2d8097 P d3c69cd2c012414f8d19e14545d2ebb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3c69cd2c012414f8d19e14545d2ebb9, State: Initialized, Role: FOLLOWER
I20240502 14:00:55.203490 3858 raft_consensus.cc:348] T 94ff128e9466454c9919d9ae6c2d8097 P 9cd2dca60ac5419f85ac80cb4d700351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e80272573649d7bab270010b4bddad" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 33541 } } peers { permanent_uuid: "d3c69cd2c012414f8d19e14545d2ebb9" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41089 } } peers { permanent_uuid: "9cd2dca60ac5419f85ac80cb4d700351" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34781 } }
I20240502 14:00:55.203523 3856 consensus_queue.cc:260] T 94ff128e9466454c9919d9ae6c2d8097 P d3c69cd2c012414f8d19e14545d2ebb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e80272573649d7bab270010b4bddad" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 33541 } } peers { permanent_uuid: "d3c69cd2c012414f8d19e14545d2ebb9" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41089 } } peers { permanent_uuid: "9cd2dca60ac5419f85ac80cb4d700351" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34781 } }
I20240502 14:00:55.203645 3858 raft_consensus.cc:374] T 94ff128e9466454c9919d9ae6c2d8097 P 9cd2dca60ac5419f85ac80cb4d700351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:55.203675 3858 raft_consensus.cc:729] T 94ff128e9466454c9919d9ae6c2d8097 P 9cd2dca60ac5419f85ac80cb4d700351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cd2dca60ac5419f85ac80cb4d700351, State: Initialized, Role: FOLLOWER
I20240502 14:00:55.203682 3857 raft_consensus.cc:348] T 94ff128e9466454c9919d9ae6c2d8097 P 44e80272573649d7bab270010b4bddad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e80272573649d7bab270010b4bddad" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 33541 } } peers { permanent_uuid: "d3c69cd2c012414f8d19e14545d2ebb9" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41089 } } peers { permanent_uuid: "9cd2dca60ac5419f85ac80cb4d700351" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34781 } }
I20240502 14:00:55.203821 3858 consensus_queue.cc:260] T 94ff128e9466454c9919d9ae6c2d8097 P 9cd2dca60ac5419f85ac80cb4d700351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e80272573649d7bab270010b4bddad" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 33541 } } peers { permanent_uuid: "d3c69cd2c012414f8d19e14545d2ebb9" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41089 } } peers { permanent_uuid: "9cd2dca60ac5419f85ac80cb4d700351" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34781 } }
I20240502 14:00:55.203933 3856 ts_tablet_manager.cc:1398] T 94ff128e9466454c9919d9ae6c2d8097 P d3c69cd2c012414f8d19e14545d2ebb9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:55.203886 3857 raft_consensus.cc:374] T 94ff128e9466454c9919d9ae6c2d8097 P 44e80272573649d7bab270010b4bddad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:55.204057 3857 raft_consensus.cc:729] T 94ff128e9466454c9919d9ae6c2d8097 P 44e80272573649d7bab270010b4bddad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44e80272573649d7bab270010b4bddad, State: Initialized, Role: FOLLOWER
I20240502 14:00:55.204174 3858 ts_tablet_manager.cc:1398] T 94ff128e9466454c9919d9ae6c2d8097 P 9cd2dca60ac5419f85ac80cb4d700351: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:55.204212 3857 consensus_queue.cc:260] T 94ff128e9466454c9919d9ae6c2d8097 P 44e80272573649d7bab270010b4bddad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e80272573649d7bab270010b4bddad" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 33541 } } peers { permanent_uuid: "d3c69cd2c012414f8d19e14545d2ebb9" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41089 } } peers { permanent_uuid: "9cd2dca60ac5419f85ac80cb4d700351" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34781 } }
I20240502 14:00:55.204922 3857 ts_tablet_manager.cc:1398] T 94ff128e9466454c9919d9ae6c2d8097 P 44e80272573649d7bab270010b4bddad: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:55.457551 3863 raft_consensus.cc:482] T 94ff128e9466454c9919d9ae6c2d8097 P 9cd2dca60ac5419f85ac80cb4d700351 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:55.457684 3863 raft_consensus.cc:504] T 94ff128e9466454c9919d9ae6c2d8097 P 9cd2dca60ac5419f85ac80cb4d700351 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e80272573649d7bab270010b4bddad" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 33541 } } peers { permanent_uuid: "d3c69cd2c012414f8d19e14545d2ebb9" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41089 } } peers { permanent_uuid: "9cd2dca60ac5419f85ac80cb4d700351" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34781 } }
I20240502 14:00:55.458204 3863 leader_election.cc:288] T 94ff128e9466454c9919d9ae6c2d8097 P 9cd2dca60ac5419f85ac80cb4d700351 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44e80272573649d7bab270010b4bddad (127.0.145.67:33541), d3c69cd2c012414f8d19e14545d2ebb9 (127.0.145.66:41089)
I20240502 14:00:55.461416 3809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94ff128e9466454c9919d9ae6c2d8097" candidate_uuid: "9cd2dca60ac5419f85ac80cb4d700351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e80272573649d7bab270010b4bddad" is_pre_election: true
I20240502 14:00:55.461437 3734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94ff128e9466454c9919d9ae6c2d8097" candidate_uuid: "9cd2dca60ac5419f85ac80cb4d700351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3c69cd2c012414f8d19e14545d2ebb9" is_pre_election: true
I20240502 14:00:55.461544 3734 raft_consensus.cc:2454] T 94ff128e9466454c9919d9ae6c2d8097 P d3c69cd2c012414f8d19e14545d2ebb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cd2dca60ac5419f85ac80cb4d700351 in term 0.
I20240502 14:00:55.461544 3809 raft_consensus.cc:2454] T 94ff128e9466454c9919d9ae6c2d8097 P 44e80272573649d7bab270010b4bddad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cd2dca60ac5419f85ac80cb4d700351 in term 0.
I20240502 14:00:55.461851 3624 leader_election.cc:302] T 94ff128e9466454c9919d9ae6c2d8097 P 9cd2dca60ac5419f85ac80cb4d700351 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cd2dca60ac5419f85ac80cb4d700351, d3c69cd2c012414f8d19e14545d2ebb9; no voters:
I20240502 14:00:55.462010 3863 raft_consensus.cc:2781] T 94ff128e9466454c9919d9ae6c2d8097 P 9cd2dca60ac5419f85ac80cb4d700351 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:55.462055 3863 raft_consensus.cc:482] T 94ff128e9466454c9919d9ae6c2d8097 P 9cd2dca60ac5419f85ac80cb4d700351 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:55.462081 3863 raft_consensus.cc:3037] T 94ff128e9466454c9919d9ae6c2d8097 P 9cd2dca60ac5419f85ac80cb4d700351 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:55.462877 3863 raft_consensus.cc:504] T 94ff128e9466454c9919d9ae6c2d8097 P 9cd2dca60ac5419f85ac80cb4d700351 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e80272573649d7bab270010b4bddad" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 33541 } } peers { permanent_uuid: "d3c69cd2c012414f8d19e14545d2ebb9" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41089 } } peers { permanent_uuid: "9cd2dca60ac5419f85ac80cb4d700351" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34781 } }
I20240502 14:00:55.463177 3863 leader_election.cc:288] T 94ff128e9466454c9919d9ae6c2d8097 P 9cd2dca60ac5419f85ac80cb4d700351 [CANDIDATE]: Term 1 election: Requested vote from peers 44e80272573649d7bab270010b4bddad (127.0.145.67:33541), d3c69cd2c012414f8d19e14545d2ebb9 (127.0.145.66:41089)
I20240502 14:00:55.463430 3734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94ff128e9466454c9919d9ae6c2d8097" candidate_uuid: "9cd2dca60ac5419f85ac80cb4d700351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3c69cd2c012414f8d19e14545d2ebb9"
I20240502 14:00:55.463430 3809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94ff128e9466454c9919d9ae6c2d8097" candidate_uuid: "9cd2dca60ac5419f85ac80cb4d700351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e80272573649d7bab270010b4bddad"
I20240502 14:00:55.463560 3809 raft_consensus.cc:3037] T 94ff128e9466454c9919d9ae6c2d8097 P 44e80272573649d7bab270010b4bddad [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:55.463580 3734 raft_consensus.cc:3037] T 94ff128e9466454c9919d9ae6c2d8097 P d3c69cd2c012414f8d19e14545d2ebb9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:55.464505 3734 raft_consensus.cc:2454] T 94ff128e9466454c9919d9ae6c2d8097 P d3c69cd2c012414f8d19e14545d2ebb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cd2dca60ac5419f85ac80cb4d700351 in term 1.
I20240502 14:00:55.464505 3809 raft_consensus.cc:2454] T 94ff128e9466454c9919d9ae6c2d8097 P 44e80272573649d7bab270010b4bddad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cd2dca60ac5419f85ac80cb4d700351 in term 1.
I20240502 14:00:55.464821 3624 leader_election.cc:302] T 94ff128e9466454c9919d9ae6c2d8097 P 9cd2dca60ac5419f85ac80cb4d700351 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cd2dca60ac5419f85ac80cb4d700351, d3c69cd2c012414f8d19e14545d2ebb9; no voters:
I20240502 14:00:55.464982 3863 raft_consensus.cc:2781] T 94ff128e9466454c9919d9ae6c2d8097 P 9cd2dca60ac5419f85ac80cb4d700351 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:55.465162 3863 raft_consensus.cc:686] T 94ff128e9466454c9919d9ae6c2d8097 P 9cd2dca60ac5419f85ac80cb4d700351 [term 1 LEADER]: Becoming Leader. State: Replica: 9cd2dca60ac5419f85ac80cb4d700351, State: Running, Role: LEADER
I20240502 14:00:55.465307 3863 consensus_queue.cc:237] T 94ff128e9466454c9919d9ae6c2d8097 P 9cd2dca60ac5419f85ac80cb4d700351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e80272573649d7bab270010b4bddad" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 33541 } } peers { permanent_uuid: "d3c69cd2c012414f8d19e14545d2ebb9" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41089 } } peers { permanent_uuid: "9cd2dca60ac5419f85ac80cb4d700351" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34781 } }
I20240502 14:00:55.466682 3555 catalog_manager.cc:5484] T 94ff128e9466454c9919d9ae6c2d8097 P 9cd2dca60ac5419f85ac80cb4d700351 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cd2dca60ac5419f85ac80cb4d700351 (127.0.145.65). New cstate: current_term: 1 leader_uuid: "9cd2dca60ac5419f85ac80cb4d700351" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e80272573649d7bab270010b4bddad" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 33541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3c69cd2c012414f8d19e14545d2ebb9" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cd2dca60ac5419f85ac80cb4d700351" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 34781 } health_report { overall_health: HEALTHY } } }
I20240502 14:00:55.904855 3863 consensus_queue.cc:1035] T 94ff128e9466454c9919d9ae6c2d8097 P 9cd2dca60ac5419f85ac80cb4d700351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44e80272573649d7bab270010b4bddad" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 33541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:55.907568 3867 consensus_queue.cc:1035] T 94ff128e9466454c9919d9ae6c2d8097 P 9cd2dca60ac5419f85ac80cb4d700351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3c69cd2c012414f8d19e14545d2ebb9" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:56.036350 3609 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:00:56.036660 3609 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:00:56.036738 3609 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:00:56.037189 3604 placement_policy_util.cc:407] tablet 94ff128e9466454c9919d9ae6c2d8097: detected majority of replicas (2 of 3) at location L0
I20240502 14:00:57.038944 3604 placement_policy_util.cc:407] tablet 94ff128e9466454c9919d9ae6c2d8097: detected majority of replicas (2 of 3) at location L0
I20240502 14:00:58.037022 3609 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:00:58.037302 3609 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:00:58.037353 3609 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:00:58.040341 3604 placement_policy_util.cc:407] tablet 94ff128e9466454c9919d9ae6c2d8097: detected majority of replicas (2 of 3) at location L0
I20240502 14:00:59.041743 3604 placement_policy_util.cc:407] tablet 94ff128e9466454c9919d9ae6c2d8097: detected majority of replicas (2 of 3) at location L0
I20240502 14:01:00.037652 3609 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:01:00.037927 3609 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:01:00.037981 3609 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:01:00.043128 3604 placement_policy_util.cc:407] tablet 94ff128e9466454c9919d9ae6c2d8097: detected majority of replicas (2 of 3) at location L0
I20240502 14:01:01.044557 3604 placement_policy_util.cc:407] tablet 94ff128e9466454c9919d9ae6c2d8097: detected majority of replicas (2 of 3) at location L0
I20240502 14:01:02.038251 3609 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:01:02.038527 3609 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:01:02.038579 3609 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:01:02.045923 3604 placement_policy_util.cc:407] tablet 94ff128e9466454c9919d9ae6c2d8097: detected majority of replicas (2 of 3) at location L0
I20240502 14:01:03.047313 3604 placement_policy_util.cc:407] tablet 94ff128e9466454c9919d9ae6c2d8097: detected majority of replicas (2 of 3) at location L0
I20240502 14:01:04.038851 3609 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:01:04.039134 3609 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:01:04.039189 3609 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:01:04.048738 3604 placement_policy_util.cc:407] tablet 94ff128e9466454c9919d9ae6c2d8097: detected majority of replicas (2 of 3) at location L0
I20240502 14:01:05.050144 3604 placement_policy_util.cc:407] tablet 94ff128e9466454c9919d9ae6c2d8097: detected majority of replicas (2 of 3) at location L0
I20240502 14:01:06.039469 3609 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:01:06.039736 3609 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:01:06.039786 3609 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:01:06.051460 3604 placement_policy_util.cc:407] tablet 94ff128e9466454c9919d9ae6c2d8097: detected majority of replicas (2 of 3) at location L0
I20240502 14:01:06.529899 581 tablet_server.cc:178] TabletServer@127.0.145.65:0 shutting down...
I20240502 14:01:06.533133 581 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:01:06.533317 581 tablet_replica.cc:330] T 94ff128e9466454c9919d9ae6c2d8097 P 9cd2dca60ac5419f85ac80cb4d700351: stopping tablet replica
I20240502 14:01:06.533432 581 raft_consensus.cc:2229] T 94ff128e9466454c9919d9ae6c2d8097 P 9cd2dca60ac5419f85ac80cb4d700351 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:01:06.533640 581 raft_consensus.cc:2258] T 94ff128e9466454c9919d9ae6c2d8097 P 9cd2dca60ac5419f85ac80cb4d700351 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:06.546061 581 tablet_server.cc:194] TabletServer@127.0.145.65:0 shutdown complete.
I20240502 14:01:06.547782 581 tablet_server.cc:178] TabletServer@127.0.145.66:0 shutting down...
I20240502 14:01:06.550635 581 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:01:06.550786 581 tablet_replica.cc:330] T 94ff128e9466454c9919d9ae6c2d8097 P d3c69cd2c012414f8d19e14545d2ebb9: stopping tablet replica
I20240502 14:01:06.550869 581 raft_consensus.cc:2229] T 94ff128e9466454c9919d9ae6c2d8097 P d3c69cd2c012414f8d19e14545d2ebb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:06.550961 581 raft_consensus.cc:2258] T 94ff128e9466454c9919d9ae6c2d8097 P d3c69cd2c012414f8d19e14545d2ebb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:06.562950 581 tablet_server.cc:194] TabletServer@127.0.145.66:0 shutdown complete.
I20240502 14:01:06.564584 581 tablet_server.cc:178] TabletServer@127.0.145.67:0 shutting down...
I20240502 14:01:06.567487 581 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:01:06.567634 581 tablet_replica.cc:330] T 94ff128e9466454c9919d9ae6c2d8097 P 44e80272573649d7bab270010b4bddad: stopping tablet replica
I20240502 14:01:06.567731 581 raft_consensus.cc:2229] T 94ff128e9466454c9919d9ae6c2d8097 P 44e80272573649d7bab270010b4bddad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:06.567822 581 raft_consensus.cc:2258] T 94ff128e9466454c9919d9ae6c2d8097 P 44e80272573649d7bab270010b4bddad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:06.579813 581 tablet_server.cc:194] TabletServer@127.0.145.67:0 shutdown complete.
I20240502 14:01:06.581360 581 master.cc:537] Master@127.0.145.126:34985 shutting down...
I20240502 14:01:06.584192 581 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3aa40935fdaa4d0dbd4066fc61abb556 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:01:06.584312 581 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3aa40935fdaa4d0dbd4066fc61abb556 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:06.584365 581 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3aa40935fdaa4d0dbd4066fc61abb556: stopping tablet replica
I20240502 14:01:06.596134 581 master.cc:552] Master@127.0.145.126:34985 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (12604 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
I20240502 14:01:06.601995 581 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.145.126:37069
I20240502 14:01:06.602283 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 14:01:06.604374 581 server_base.cc:1034] running on GCE node
W20240502 14:01:06.604715 3878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:06.605160 3881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:06.605579 3879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:01:06.605744 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:01:06.605789 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:01:06.605803 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658466605802 us; error 0 us; skew 500 ppm
I20240502 14:01:06.606663 581 webserver.cc:460] Webserver started at http://127.0.145.126:40473/ using document root <none> and password file <none>
I20240502 14:01:06.606871 581 fs_manager.cc:359] Metadata directory not provided
I20240502 14:01:06.606936 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:01:06.607013 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:01:06.607529 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/master-0-root/instance:
uuid: "e3566aa929e84f6799719a8c8ce33a99"
format_stamp: "Formatted at 2024-05-02 14:01:06 on dist-test-slave-1z5j"
I20240502 14:01:06.609097 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 14:01:06.609961 3886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:06.610204 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:06.610267 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/master-0-root
uuid: "e3566aa929e84f6799719a8c8ce33a99"
format_stamp: "Formatted at 2024-05-02 14:01:06 on dist-test-slave-1z5j"
I20240502 14:01:06.610337 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:01:06.616684 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:01:06.616971 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:01:06.620483 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.126:37069
I20240502 14:01:06.621943 3938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.126:37069 every 8 connection(s)
I20240502 14:01:06.622309 3939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:06.623903 3939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3566aa929e84f6799719a8c8ce33a99: Bootstrap starting.
I20240502 14:01:06.624552 3939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3566aa929e84f6799719a8c8ce33a99: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:06.625471 3939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3566aa929e84f6799719a8c8ce33a99: No bootstrap required, opened a new log
I20240502 14:01:06.625784 3939 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3566aa929e84f6799719a8c8ce33a99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3566aa929e84f6799719a8c8ce33a99" member_type: VOTER }
I20240502 14:01:06.625877 3939 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3566aa929e84f6799719a8c8ce33a99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:06.625916 3939 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3566aa929e84f6799719a8c8ce33a99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3566aa929e84f6799719a8c8ce33a99, State: Initialized, Role: FOLLOWER
I20240502 14:01:06.626006 3939 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3566aa929e84f6799719a8c8ce33a99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3566aa929e84f6799719a8c8ce33a99" member_type: VOTER }
I20240502 14:01:06.626060 3939 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3566aa929e84f6799719a8c8ce33a99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:01:06.626084 3939 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3566aa929e84f6799719a8c8ce33a99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:01:06.626116 3939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3566aa929e84f6799719a8c8ce33a99 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:06.626698 3939 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3566aa929e84f6799719a8c8ce33a99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3566aa929e84f6799719a8c8ce33a99" member_type: VOTER }
I20240502 14:01:06.626824 3939 leader_election.cc:302] T 00000000000000000000000000000000 P e3566aa929e84f6799719a8c8ce33a99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3566aa929e84f6799719a8c8ce33a99; no voters:
I20240502 14:01:06.626951 3939 leader_election.cc:288] T 00000000000000000000000000000000 P e3566aa929e84f6799719a8c8ce33a99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:01:06.627065 3942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3566aa929e84f6799719a8c8ce33a99 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:06.627229 3942 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3566aa929e84f6799719a8c8ce33a99 [term 1 LEADER]: Becoming Leader. State: Replica: e3566aa929e84f6799719a8c8ce33a99, State: Running, Role: LEADER
I20240502 14:01:06.627383 3939 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3566aa929e84f6799719a8c8ce33a99 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:01:06.627348 3942 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3566aa929e84f6799719a8c8ce33a99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3566aa929e84f6799719a8c8ce33a99" member_type: VOTER }
I20240502 14:01:06.627738 3943 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3566aa929e84f6799719a8c8ce33a99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3566aa929e84f6799719a8c8ce33a99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3566aa929e84f6799719a8c8ce33a99" member_type: VOTER } }
I20240502 14:01:06.627756 3944 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3566aa929e84f6799719a8c8ce33a99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3566aa929e84f6799719a8c8ce33a99. Latest consensus state: current_term: 1 leader_uuid: "e3566aa929e84f6799719a8c8ce33a99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3566aa929e84f6799719a8c8ce33a99" member_type: VOTER } }
I20240502 14:01:06.627869 3943 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3566aa929e84f6799719a8c8ce33a99 [sys.catalog]: This master's current role is: LEADER
I20240502 14:01:06.627947 3944 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3566aa929e84f6799719a8c8ce33a99 [sys.catalog]: This master's current role is: LEADER
I20240502 14:01:06.628407 3948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:01:06.629276 3948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:01:06.629505 581 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 14:01:06.630987 3948 catalog_manager.cc:1347] Generated new cluster ID: 899451c6f16f40088d3bb9e2080a48f4
I20240502 14:01:06.631044 3948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:01:06.645809 3948 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:01:06.646333 3948 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:01:06.653003 3948 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e3566aa929e84f6799719a8c8ce33a99: Generated new TSK 0
I20240502 14:01:06.653151 3948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:01:06.661726 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:01:06.663915 3963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:06.664397 3961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:06.664680 3960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:01:06.665114 581 server_base.cc:1034] running on GCE node
I20240502 14:01:06.665247 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:01:06.665282 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:01:06.665300 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658466665299 us; error 0 us; skew 500 ppm
I20240502 14:01:06.666126 581 webserver.cc:460] Webserver started at http://127.0.145.65:32859/ using document root <none> and password file <none>
I20240502 14:01:06.666261 581 fs_manager.cc:359] Metadata directory not provided
I20240502 14:01:06.666308 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:01:06.666379 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:01:06.666719 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-0-root/instance:
uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
format_stamp: "Formatted at 2024-05-02 14:01:06 on dist-test-slave-1z5j"
I20240502 14:01:06.668026 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 14:01:06.668818 3968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:06.669018 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:06.669080 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-0-root
uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
format_stamp: "Formatted at 2024-05-02 14:01:06 on dist-test-slave-1z5j"
I20240502 14:01:06.669143 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:01:06.687011 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:01:06.687320 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:01:06.687564 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:01:06.687986 581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:01:06.688022 581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:06.688061 581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:01:06.688089 581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:06.691888 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.65:46667
I20240502 14:01:06.691927 4031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.65:46667 every 8 connection(s)
I20240502 14:01:06.692610 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:01:06.694612 4036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:06.695149 4037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:06.695608 4039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:01:06.696148 581 server_base.cc:1034] running on GCE node
I20240502 14:01:06.696297 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:01:06.696342 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:01:06.696357 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658466696357 us; error 0 us; skew 500 ppm
I20240502 14:01:06.697186 4032 heartbeater.cc:347] Connected to a master server at 127.0.145.126:37069
I20240502 14:01:06.697270 4032 heartbeater.cc:464] Registering TS with master...
I20240502 14:01:06.697345 581 webserver.cc:460] Webserver started at http://127.0.145.66:39009/ using document root <none> and password file <none>
I20240502 14:01:06.697489 581 fs_manager.cc:359] Metadata directory not provided
I20240502 14:01:06.697497 4032 heartbeater.cc:511] Master 127.0.145.126:37069 requested a full tablet report, sending...
I20240502 14:01:06.697574 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:01:06.697643 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:01:06.698020 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-1-root/instance:
uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
format_stamp: "Formatted at 2024-05-02 14:01:06 on dist-test-slave-1z5j"
I20240502 14:01:06.698125 3903 ts_manager.cc:194] Registered new tserver with Master: bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667)
I20240502 14:01:06.699100 3903 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40812
I20240502 14:01:06.699609 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:06.700454 4044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:06.700675 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:06.700732 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-1-root
uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
format_stamp: "Formatted at 2024-05-02 14:01:06 on dist-test-slave-1z5j"
I20240502 14:01:06.700816 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:01:06.708007 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:01:06.708248 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:01:06.708464 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:01:06.708879 581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:01:06.708916 581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:06.708953 581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:01:06.708984 581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:06.712800 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.66:35251
I20240502 14:01:06.712836 4107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.66:35251 every 8 connection(s)
I20240502 14:01:06.713522 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:01:06.715267 4111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:06.715873 4114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:06.716235 4112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:01:06.716637 581 server_base.cc:1034] running on GCE node
I20240502 14:01:06.716822 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240502 14:01:06.716867 4108 heartbeater.cc:347] Connected to a master server at 127.0.145.126:37069
W20240502 14:01:06.716868 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:01:06.716954 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658466716954 us; error 0 us; skew 500 ppm
I20240502 14:01:06.716971 4108 heartbeater.cc:464] Registering TS with master...
I20240502 14:01:06.717156 4108 heartbeater.cc:511] Master 127.0.145.126:37069 requested a full tablet report, sending...
I20240502 14:01:06.717715 3903 ts_manager.cc:194] Registered new tserver with Master: 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251)
I20240502 14:01:06.717825 581 webserver.cc:460] Webserver started at http://127.0.145.67:38817/ using document root <none> and password file <none>
I20240502 14:01:06.717954 581 fs_manager.cc:359] Metadata directory not provided
I20240502 14:01:06.718008 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:01:06.718073 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:01:06.718343 3903 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40828
I20240502 14:01:06.718457 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-2-root/instance:
uuid: "df655d53530449fc85de33de1c229815"
format_stamp: "Formatted at 2024-05-02 14:01:06 on dist-test-slave-1z5j"
I20240502 14:01:06.719775 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 14:01:06.720518 4119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:06.720719 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:06.720798 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-2-root
uuid: "df655d53530449fc85de33de1c229815"
format_stamp: "Formatted at 2024-05-02 14:01:06 on dist-test-slave-1z5j"
I20240502 14:01:06.720856 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:01:06.731379 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:01:06.731665 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:01:06.731916 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:01:06.732276 581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:01:06.732311 581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:06.732347 581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:01:06.732374 581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:06.736402 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.67:34309
I20240502 14:01:06.736436 4182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.67:34309 every 8 connection(s)
I20240502 14:01:06.739938 4183 heartbeater.cc:347] Connected to a master server at 127.0.145.126:37069
I20240502 14:01:06.740008 4183 heartbeater.cc:464] Registering TS with master...
I20240502 14:01:06.740175 4183 heartbeater.cc:511] Master 127.0.145.126:37069 requested a full tablet report, sending...
I20240502 14:01:06.740679 3903 ts_manager.cc:194] Registered new tserver with Master: df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:06.741169 3903 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40842
I20240502 14:01:06.741243 581 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004471493s
I20240502 14:01:07.700004 4032 heartbeater.cc:503] Master 127.0.145.126:37069 was elected leader, sending a full tablet report...
I20240502 14:01:07.719230 4108 heartbeater.cc:503] Master 127.0.145.126:37069 was elected leader, sending a full tablet report...
I20240502 14:01:07.741930 4183 heartbeater.cc:503] Master 127.0.145.126:37069 was elected leader, sending a full tablet report...
I20240502 14:01:07.766674 581 test_util.cc:268] Using random seed: 149021072
I20240502 14:01:07.773662 3903 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40854:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu""\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:01:07.774183 3903 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:01:07.786743 4147 tablet_service.cc:1467] Processing CreateTablet for tablet fb4c3dcde8764e17bb084cb37fef706f (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240502 14:01:07.787079 4147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb4c3dcde8764e17bb084cb37fef706f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.787200 4146 tablet_service.cc:1467] Processing CreateTablet for tablet dba8942501c440b18f71c7106878d0c4 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240502 14:01:07.787459 4146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dba8942501c440b18f71c7106878d0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.787737 3994 tablet_service.cc:1467] Processing CreateTablet for tablet e6ca4998e3634ba789210e0c072e29b0 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240502 14:01:07.787978 3994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6ca4998e3634ba789210e0c072e29b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.789402 4203 tablet_bootstrap.cc:492] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815: Bootstrap starting.
I20240502 14:01:07.789541 4141 tablet_service.cc:1467] Processing CreateTablet for tablet 144739b71bf6474d90399816120b3b59 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240502 14:01:07.789726 4144 tablet_service.cc:1467] Processing CreateTablet for tablet 46b4a50dee854f62aac3a02889525b40 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240502 14:01:07.789790 4141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 144739b71bf6474d90399816120b3b59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.789950 4144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46b4a50dee854f62aac3a02889525b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.790189 3994 tablet_service.cc:1467] Processing CreateTablet for tablet 3d0d4aeaec264d04bae2c34e6edc6ec7 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240502 14:01:07.790282 4203 tablet_bootstrap.cc:654] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.790406 3994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d0d4aeaec264d04bae2c34e6edc6ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.791621 4143 tablet_service.cc:1467] Processing CreateTablet for tablet a4d0e9d6d8e7445a9367e0f3a31c8748 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240502 14:01:07.791790 4143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4d0e9d6d8e7445a9367e0f3a31c8748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.792054 3994 tablet_service.cc:1467] Processing CreateTablet for tablet b5172e93df3a4fb5a70188b0507639e6 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240502 14:01:07.792258 3994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5172e93df3a4fb5a70188b0507639e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.792918 4203 tablet_bootstrap.cc:492] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815: No bootstrap required, opened a new log
I20240502 14:01:07.793051 3987 tablet_service.cc:1467] Processing CreateTablet for tablet 7ac79a5b1f514bdcb36e0db4ff6e3a59 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240502 14:01:07.793334 3987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ac79a5b1f514bdcb36e0db4ff6e3a59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.793305 4142 tablet_service.cc:1467] Processing CreateTablet for tablet 975becfb8df0430fa8c748a58dcbcfb1 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240502 14:01:07.793534 4142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 975becfb8df0430fa8c748a58dcbcfb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.793059 4203 ts_tablet_manager.cc:1367] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:01:07.795104 4203 raft_consensus.cc:348] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } }
I20240502 14:01:07.795130 4140 tablet_service.cc:1467] Processing CreateTablet for tablet c965f0e11bc1433f8ccdc20c98b4b4b9 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240502 14:01:07.795192 4203 raft_consensus.cc:374] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.795221 4203 raft_consensus.cc:729] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df655d53530449fc85de33de1c229815, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.795317 4140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c965f0e11bc1433f8ccdc20c98b4b4b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.795342 4203 consensus_queue.cc:260] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } }
I20240502 14:01:07.795680 4203 ts_tablet_manager.cc:1398] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:07.795858 4203 tablet_bootstrap.cc:492] T dba8942501c440b18f71c7106878d0c4 P df655d53530449fc85de33de1c229815: Bootstrap starting.
I20240502 14:01:07.796609 4203 tablet_bootstrap.cc:654] T dba8942501c440b18f71c7106878d0c4 P df655d53530449fc85de33de1c229815: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.797022 4139 tablet_service.cc:1467] Processing CreateTablet for tablet 7908edf8bbe8419ab21d0c93cd2a3b35 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240502 14:01:07.797389 4139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7908edf8bbe8419ab21d0c93cd2a3b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.797358 3996 tablet_service.cc:1467] Processing CreateTablet for tablet fb4c3dcde8764e17bb084cb37fef706f (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240502 14:01:07.797570 3996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb4c3dcde8764e17bb084cb37fef706f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.797717 4203 tablet_bootstrap.cc:492] T dba8942501c440b18f71c7106878d0c4 P df655d53530449fc85de33de1c229815: No bootstrap required, opened a new log
I20240502 14:01:07.797793 4203 ts_tablet_manager.cc:1367] T dba8942501c440b18f71c7106878d0c4 P df655d53530449fc85de33de1c229815: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:01:07.798264 4203 raft_consensus.cc:348] T dba8942501c440b18f71c7106878d0c4 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.798342 4203 raft_consensus.cc:374] T dba8942501c440b18f71c7106878d0c4 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.798362 4203 raft_consensus.cc:729] T dba8942501c440b18f71c7106878d0c4 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df655d53530449fc85de33de1c229815, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.798461 4203 consensus_queue.cc:260] T dba8942501c440b18f71c7106878d0c4 P df655d53530449fc85de33de1c229815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.798679 4203 ts_tablet_manager.cc:1398] T dba8942501c440b18f71c7106878d0c4 P df655d53530449fc85de33de1c229815: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:01:07.798842 4203 tablet_bootstrap.cc:492] T 46b4a50dee854f62aac3a02889525b40 P df655d53530449fc85de33de1c229815: Bootstrap starting.
I20240502 14:01:07.799305 3995 tablet_service.cc:1467] Processing CreateTablet for tablet dba8942501c440b18f71c7106878d0c4 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240502 14:01:07.799542 3995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dba8942501c440b18f71c7106878d0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.799765 4203 tablet_bootstrap.cc:654] T 46b4a50dee854f62aac3a02889525b40 P df655d53530449fc85de33de1c229815: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.800698 4203 tablet_bootstrap.cc:492] T 46b4a50dee854f62aac3a02889525b40 P df655d53530449fc85de33de1c229815: No bootstrap required, opened a new log
I20240502 14:01:07.800823 4203 ts_tablet_manager.cc:1367] T 46b4a50dee854f62aac3a02889525b40 P df655d53530449fc85de33de1c229815: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:01:07.801182 4203 raft_consensus.cc:348] T 46b4a50dee854f62aac3a02889525b40 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.801261 4203 raft_consensus.cc:374] T 46b4a50dee854f62aac3a02889525b40 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.801290 4203 raft_consensus.cc:729] T 46b4a50dee854f62aac3a02889525b40 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df655d53530449fc85de33de1c229815, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.801451 4203 consensus_queue.cc:260] T 46b4a50dee854f62aac3a02889525b40 P df655d53530449fc85de33de1c229815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.801476 3993 tablet_service.cc:1467] Processing CreateTablet for tablet 46b4a50dee854f62aac3a02889525b40 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240502 14:01:07.801690 3993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46b4a50dee854f62aac3a02889525b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.801719 3992 tablet_service.cc:1467] Processing CreateTablet for tablet a4d0e9d6d8e7445a9367e0f3a31c8748 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240502 14:01:07.801935 3992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4d0e9d6d8e7445a9367e0f3a31c8748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.803248 4203 ts_tablet_manager.cc:1398] T 46b4a50dee854f62aac3a02889525b40 P df655d53530449fc85de33de1c229815: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:07.803450 4203 tablet_bootstrap.cc:492] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815: Bootstrap starting.
I20240502 14:01:07.803483 3991 tablet_service.cc:1467] Processing CreateTablet for tablet 975becfb8df0430fa8c748a58dcbcfb1 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240502 14:01:07.803716 3991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 975becfb8df0430fa8c748a58dcbcfb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.804141 4203 tablet_bootstrap.cc:654] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.805022 3988 tablet_service.cc:1467] Processing CreateTablet for tablet 7908edf8bbe8419ab21d0c93cd2a3b35 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240502 14:01:07.805243 3988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7908edf8bbe8419ab21d0c93cd2a3b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.805332 4203 tablet_bootstrap.cc:492] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815: No bootstrap required, opened a new log
I20240502 14:01:07.805408 4203 ts_tablet_manager.cc:1367] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:07.805768 3990 tablet_service.cc:1467] Processing CreateTablet for tablet 144739b71bf6474d90399816120b3b59 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240502 14:01:07.805909 3989 tablet_service.cc:1467] Processing CreateTablet for tablet c965f0e11bc1433f8ccdc20c98b4b4b9 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240502 14:01:07.806007 3990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 144739b71bf6474d90399816120b3b59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.789561 4145 tablet_service.cc:1467] Processing CreateTablet for tablet e6ca4998e3634ba789210e0c072e29b0 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240502 14:01:07.806198 3989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c965f0e11bc1433f8ccdc20c98b4b4b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.806253 4145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6ca4998e3634ba789210e0c072e29b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.805760 4203 raft_consensus.cc:348] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.807816 4203 raft_consensus.cc:374] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.807858 4203 raft_consensus.cc:729] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df655d53530449fc85de33de1c229815, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.807957 4203 consensus_queue.cc:260] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.808218 4203 ts_tablet_manager.cc:1398] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:01:07.808379 4203 tablet_bootstrap.cc:492] T a4d0e9d6d8e7445a9367e0f3a31c8748 P df655d53530449fc85de33de1c229815: Bootstrap starting.
I20240502 14:01:07.808540 4139 tablet_service.cc:1467] Processing CreateTablet for tablet 3d0d4aeaec264d04bae2c34e6edc6ec7 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240502 14:01:07.808751 4139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d0d4aeaec264d04bae2c34e6edc6ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.809170 4203 tablet_bootstrap.cc:654] T a4d0e9d6d8e7445a9367e0f3a31c8748 P df655d53530449fc85de33de1c229815: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.809517 4142 tablet_service.cc:1467] Processing CreateTablet for tablet b5172e93df3a4fb5a70188b0507639e6 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240502 14:01:07.809727 4142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5172e93df3a4fb5a70188b0507639e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.810663 4145 tablet_service.cc:1467] Processing CreateTablet for tablet 7ac79a5b1f514bdcb36e0db4ff6e3a59 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240502 14:01:07.810865 4145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ac79a5b1f514bdcb36e0db4ff6e3a59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.811008 4203 tablet_bootstrap.cc:492] T a4d0e9d6d8e7445a9367e0f3a31c8748 P df655d53530449fc85de33de1c229815: No bootstrap required, opened a new log
I20240502 14:01:07.811190 4203 ts_tablet_manager.cc:1367] T a4d0e9d6d8e7445a9367e0f3a31c8748 P df655d53530449fc85de33de1c229815: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:01:07.811602 4203 raft_consensus.cc:348] T a4d0e9d6d8e7445a9367e0f3a31c8748 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.811736 4203 raft_consensus.cc:374] T a4d0e9d6d8e7445a9367e0f3a31c8748 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.811779 4203 raft_consensus.cc:729] T a4d0e9d6d8e7445a9367e0f3a31c8748 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df655d53530449fc85de33de1c229815, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.811897 4203 consensus_queue.cc:260] T a4d0e9d6d8e7445a9367e0f3a31c8748 P df655d53530449fc85de33de1c229815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.812139 4203 ts_tablet_manager.cc:1398] T a4d0e9d6d8e7445a9367e0f3a31c8748 P df655d53530449fc85de33de1c229815: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:01:07.812328 4203 tablet_bootstrap.cc:492] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815: Bootstrap starting.
I20240502 14:01:07.813154 4203 tablet_bootstrap.cc:654] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.813380 4204 tablet_bootstrap.cc:492] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae: Bootstrap starting.
I20240502 14:01:07.814158 4203 tablet_bootstrap.cc:492] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815: No bootstrap required, opened a new log
I20240502 14:01:07.814152 4071 tablet_service.cc:1467] Processing CreateTablet for tablet dba8942501c440b18f71c7106878d0c4 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240502 14:01:07.814280 4204 tablet_bootstrap.cc:654] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.814319 4203 ts_tablet_manager.cc:1367] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:01:07.814421 4071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dba8942501c440b18f71c7106878d0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.814651 4072 tablet_service.cc:1467] Processing CreateTablet for tablet fb4c3dcde8764e17bb084cb37fef706f (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240502 14:01:07.814698 4203 raft_consensus.cc:348] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.814788 4203 raft_consensus.cc:374] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.814816 4203 raft_consensus.cc:729] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df655d53530449fc85de33de1c229815, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.814860 4072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb4c3dcde8764e17bb084cb37fef706f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.814942 4203 consensus_queue.cc:260] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.815023 4063 tablet_service.cc:1467] Processing CreateTablet for tablet 7ac79a5b1f514bdcb36e0db4ff6e3a59 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240502 14:01:07.815232 4063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ac79a5b1f514bdcb36e0db4ff6e3a59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.815299 4203 ts_tablet_manager.cc:1398] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:01:07.815456 4203 tablet_bootstrap.cc:492] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815: Bootstrap starting.
I20240502 14:01:07.816227 4203 tablet_bootstrap.cc:654] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.816516 4071 tablet_service.cc:1467] Processing CreateTablet for tablet 3d0d4aeaec264d04bae2c34e6edc6ec7 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240502 14:01:07.816660 4072 tablet_service.cc:1467] Processing CreateTablet for tablet b5172e93df3a4fb5a70188b0507639e6 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240502 14:01:07.817000 4213 tablet_bootstrap.cc:492] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45: Bootstrap starting.
I20240502 14:01:07.817137 4064 tablet_service.cc:1467] Processing CreateTablet for tablet 7908edf8bbe8419ab21d0c93cd2a3b35 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240502 14:01:07.816716 4071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d0d4aeaec264d04bae2c34e6edc6ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.817387 4064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7908edf8bbe8419ab21d0c93cd2a3b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.817389 4203 tablet_bootstrap.cc:492] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815: No bootstrap required, opened a new log
I20240502 14:01:07.817523 4203 ts_tablet_manager.cc:1367] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:01:07.817811 4065 tablet_service.cc:1467] Processing CreateTablet for tablet c965f0e11bc1433f8ccdc20c98b4b4b9 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240502 14:01:07.817935 4203 raft_consensus.cc:348] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.818076 4203 raft_consensus.cc:374] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.817574 4072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5172e93df3a4fb5a70188b0507639e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.818153 4203 raft_consensus.cc:729] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df655d53530449fc85de33de1c229815, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.818202 4066 tablet_service.cc:1467] Processing CreateTablet for tablet 144739b71bf6474d90399816120b3b59 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240502 14:01:07.818317 4213 tablet_bootstrap.cc:654] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.818296 4203 consensus_queue.cc:260] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.818411 4066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 144739b71bf6474d90399816120b3b59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.818608 4203 ts_tablet_manager.cc:1398] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:07.818746 4203 tablet_bootstrap.cc:492] T e6ca4998e3634ba789210e0c072e29b0 P df655d53530449fc85de33de1c229815: Bootstrap starting.
I20240502 14:01:07.819260 4213 tablet_bootstrap.cc:492] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45: No bootstrap required, opened a new log
I20240502 14:01:07.819334 4213 ts_tablet_manager.cc:1367] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:01:07.819633 4203 tablet_bootstrap.cc:654] T e6ca4998e3634ba789210e0c072e29b0 P df655d53530449fc85de33de1c229815: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.819703 4213 raft_consensus.cc:348] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.819787 4213 raft_consensus.cc:374] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.819824 4213 raft_consensus.cc:729] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.819933 4213 consensus_queue.cc:260] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.820220 4213 ts_tablet_manager.cc:1398] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:07.820200 4067 tablet_service.cc:1467] Processing CreateTablet for tablet 975becfb8df0430fa8c748a58dcbcfb1 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240502 14:01:07.820369 4213 tablet_bootstrap.cc:492] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45: Bootstrap starting.
I20240502 14:01:07.820479 4067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 975becfb8df0430fa8c748a58dcbcfb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.820521 4068 tablet_service.cc:1467] Processing CreateTablet for tablet a4d0e9d6d8e7445a9367e0f3a31c8748 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240502 14:01:07.820858 4069 tablet_service.cc:1467] Processing CreateTablet for tablet 46b4a50dee854f62aac3a02889525b40 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240502 14:01:07.821175 4070 tablet_service.cc:1467] Processing CreateTablet for tablet e6ca4998e3634ba789210e0c072e29b0 (DEFAULT_TABLE table=test-workload [id=66735bdbe8fc4765af013422ce730ad0]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240502 14:01:07.821355 4065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c965f0e11bc1433f8ccdc20c98b4b4b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.821401 4213 tablet_bootstrap.cc:654] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.822424 4213 tablet_bootstrap.cc:492] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45: No bootstrap required, opened a new log
I20240502 14:01:07.822503 4213 ts_tablet_manager.cc:1367] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:01:07.822647 4068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4d0e9d6d8e7445a9367e0f3a31c8748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.822865 4213 raft_consensus.cc:348] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } }
I20240502 14:01:07.822958 4213 raft_consensus.cc:374] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.822988 4213 raft_consensus.cc:729] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.823096 4213 consensus_queue.cc:260] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } }
I20240502 14:01:07.823405 4213 ts_tablet_manager.cc:1398] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:07.823547 4213 tablet_bootstrap.cc:492] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 24fd5d6c84c1483ea8f39bb975e26e45: Bootstrap starting.
I20240502 14:01:07.823662 4204 tablet_bootstrap.cc:492] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae: No bootstrap required, opened a new log
I20240502 14:01:07.823736 4204 ts_tablet_manager.cc:1367] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240502 14:01:07.824116 4204 raft_consensus.cc:348] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.824203 4204 raft_consensus.cc:374] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.824234 4204 raft_consensus.cc:729] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.824321 4213 tablet_bootstrap.cc:654] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 24fd5d6c84c1483ea8f39bb975e26e45: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.824352 4204 consensus_queue.cc:260] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.824491 4069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46b4a50dee854f62aac3a02889525b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.824672 4204 ts_tablet_manager.cc:1398] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:07.824851 4204 tablet_bootstrap.cc:492] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P bd060dd4339f4fa5a985596bf7dec0ae: Bootstrap starting.
I20240502 14:01:07.824898 4070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6ca4998e3634ba789210e0c072e29b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:07.825686 4204 tablet_bootstrap.cc:654] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P bd060dd4339f4fa5a985596bf7dec0ae: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.826682 4204 tablet_bootstrap.cc:492] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P bd060dd4339f4fa5a985596bf7dec0ae: No bootstrap required, opened a new log
I20240502 14:01:07.826828 4204 ts_tablet_manager.cc:1367] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P bd060dd4339f4fa5a985596bf7dec0ae: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:01:07.827247 4204 raft_consensus.cc:348] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.827390 4204 raft_consensus.cc:374] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.827543 4204 raft_consensus.cc:729] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.827773 4204 consensus_queue.cc:260] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P bd060dd4339f4fa5a985596bf7dec0ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.828182 4204 ts_tablet_manager.cc:1398] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P bd060dd4339f4fa5a985596bf7dec0ae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:07.828390 4204 tablet_bootstrap.cc:492] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae: Bootstrap starting.
I20240502 14:01:07.829227 4213 tablet_bootstrap.cc:492] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 24fd5d6c84c1483ea8f39bb975e26e45: No bootstrap required, opened a new log
I20240502 14:01:07.829309 4213 ts_tablet_manager.cc:1367] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 24fd5d6c84c1483ea8f39bb975e26e45: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:01:07.829811 4204 tablet_bootstrap.cc:654] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.829794 4213 raft_consensus.cc:348] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.829963 4213 raft_consensus.cc:374] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.830001 4213 raft_consensus.cc:729] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.830112 4203 tablet_bootstrap.cc:492] T e6ca4998e3634ba789210e0c072e29b0 P df655d53530449fc85de33de1c229815: No bootstrap required, opened a new log
I20240502 14:01:07.830199 4203 ts_tablet_manager.cc:1367] T e6ca4998e3634ba789210e0c072e29b0 P df655d53530449fc85de33de1c229815: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20240502 14:01:07.830168 4213 consensus_queue.cc:260] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 24fd5d6c84c1483ea8f39bb975e26e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.830502 4213 ts_tablet_manager.cc:1398] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 24fd5d6c84c1483ea8f39bb975e26e45: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:07.830768 4213 tablet_bootstrap.cc:492] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45: Bootstrap starting.
I20240502 14:01:07.830865 4204 tablet_bootstrap.cc:492] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae: No bootstrap required, opened a new log
I20240502 14:01:07.830919 4204 ts_tablet_manager.cc:1367] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:01:07.831254 4204 raft_consensus.cc:348] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.831352 4204 raft_consensus.cc:374] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.831395 4204 raft_consensus.cc:729] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.831532 4204 consensus_queue.cc:260] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.831797 4204 ts_tablet_manager.cc:1398] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:01:07.831885 4213 tablet_bootstrap.cc:654] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.831952 4204 tablet_bootstrap.cc:492] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P bd060dd4339f4fa5a985596bf7dec0ae: Bootstrap starting.
I20240502 14:01:07.832599 4204 tablet_bootstrap.cc:654] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P bd060dd4339f4fa5a985596bf7dec0ae: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.830513 4203 raft_consensus.cc:348] T e6ca4998e3634ba789210e0c072e29b0 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.833207 4203 raft_consensus.cc:374] T e6ca4998e3634ba789210e0c072e29b0 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.833258 4203 raft_consensus.cc:729] T e6ca4998e3634ba789210e0c072e29b0 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df655d53530449fc85de33de1c229815, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.833421 4203 consensus_queue.cc:260] T e6ca4998e3634ba789210e0c072e29b0 P df655d53530449fc85de33de1c229815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.833747 4203 ts_tablet_manager.cc:1398] T e6ca4998e3634ba789210e0c072e29b0 P df655d53530449fc85de33de1c229815: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240502 14:01:07.834251 4203 tablet_bootstrap.cc:492] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815: Bootstrap starting.
I20240502 14:01:07.834432 4213 tablet_bootstrap.cc:492] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45: No bootstrap required, opened a new log
I20240502 14:01:07.834553 4213 ts_tablet_manager.cc:1367] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:01:07.835053 4203 tablet_bootstrap.cc:654] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.835078 4213 raft_consensus.cc:348] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.835198 4213 raft_consensus.cc:374] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.835250 4213 raft_consensus.cc:729] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.835541 4213 consensus_queue.cc:260] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.835664 4204 tablet_bootstrap.cc:492] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P bd060dd4339f4fa5a985596bf7dec0ae: No bootstrap required, opened a new log
I20240502 14:01:07.835721 4204 ts_tablet_manager.cc:1367] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P bd060dd4339f4fa5a985596bf7dec0ae: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240502 14:01:07.835875 4213 ts_tablet_manager.cc:1398] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:07.836047 4204 raft_consensus.cc:348] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.836133 4204 raft_consensus.cc:374] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.836162 4204 raft_consensus.cc:729] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.836226 4213 tablet_bootstrap.cc:492] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45: Bootstrap starting.
I20240502 14:01:07.836282 4204 consensus_queue.cc:260] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P bd060dd4339f4fa5a985596bf7dec0ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.836529 4204 ts_tablet_manager.cc:1398] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P bd060dd4339f4fa5a985596bf7dec0ae: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:01:07.837131 4204 tablet_bootstrap.cc:492] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae: Bootstrap starting.
I20240502 14:01:07.838232 4204 tablet_bootstrap.cc:654] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.838384 4213 tablet_bootstrap.cc:654] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.839155 4203 tablet_bootstrap.cc:492] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815: No bootstrap required, opened a new log
I20240502 14:01:07.839238 4203 ts_tablet_manager.cc:1367] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240502 14:01:07.839345 4204 tablet_bootstrap.cc:492] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae: No bootstrap required, opened a new log
I20240502 14:01:07.839427 4204 ts_tablet_manager.cc:1367] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:01:07.839627 4203 raft_consensus.cc:348] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.839701 4213 tablet_bootstrap.cc:492] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45: No bootstrap required, opened a new log
I20240502 14:01:07.839720 4203 raft_consensus.cc:374] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.839754 4203 raft_consensus.cc:729] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df655d53530449fc85de33de1c229815, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.839756 4204 raft_consensus.cc:348] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } }
I20240502 14:01:07.839866 4204 raft_consensus.cc:374] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.839905 4204 raft_consensus.cc:729] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.839913 4203 consensus_queue.cc:260] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.840006 4204 consensus_queue.cc:260] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } }
I20240502 14:01:07.840171 4203 ts_tablet_manager.cc:1398] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:01:07.840214 4204 ts_tablet_manager.cc:1398] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:01:07.840335 4204 tablet_bootstrap.cc:492] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae: Bootstrap starting.
I20240502 14:01:07.840335 4203 tablet_bootstrap.cc:492] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P df655d53530449fc85de33de1c229815: Bootstrap starting.
I20240502 14:01:07.841154 4204 tablet_bootstrap.cc:654] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.841500 4203 tablet_bootstrap.cc:654] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P df655d53530449fc85de33de1c229815: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.839776 4213 ts_tablet_manager.cc:1367] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:01:07.842288 4213 raft_consensus.cc:348] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.842392 4213 raft_consensus.cc:374] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.842425 4213 raft_consensus.cc:729] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.842518 4213 consensus_queue.cc:260] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.842730 4213 ts_tablet_manager.cc:1398] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:07.842883 4203 tablet_bootstrap.cc:492] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P df655d53530449fc85de33de1c229815: No bootstrap required, opened a new log
I20240502 14:01:07.842912 4213 tablet_bootstrap.cc:492] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45: Bootstrap starting.
I20240502 14:01:07.842962 4203 ts_tablet_manager.cc:1367] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P df655d53530449fc85de33de1c229815: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:01:07.843287 4204 tablet_bootstrap.cc:492] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae: No bootstrap required, opened a new log
I20240502 14:01:07.843286 4203 raft_consensus.cc:348] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.843405 4203 raft_consensus.cc:374] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.843426 4204 ts_tablet_manager.cc:1367] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:01:07.843439 4203 raft_consensus.cc:729] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df655d53530449fc85de33de1c229815, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.843552 4203 consensus_queue.cc:260] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P df655d53530449fc85de33de1c229815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.843710 4213 tablet_bootstrap.cc:654] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.843763 4204 raft_consensus.cc:348] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.843806 4203 ts_tablet_manager.cc:1398] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P df655d53530449fc85de33de1c229815: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:07.843859 4204 raft_consensus.cc:374] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.843899 4204 raft_consensus.cc:729] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.843984 4203 tablet_bootstrap.cc:492] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815: Bootstrap starting.
I20240502 14:01:07.844004 4204 consensus_queue.cc:260] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.844229 4204 ts_tablet_manager.cc:1398] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:07.844388 4204 tablet_bootstrap.cc:492] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae: Bootstrap starting.
I20240502 14:01:07.844851 4203 tablet_bootstrap.cc:654] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.844892 4213 tablet_bootstrap.cc:492] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45: No bootstrap required, opened a new log
I20240502 14:01:07.844962 4213 ts_tablet_manager.cc:1367] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:01:07.845333 4213 raft_consensus.cc:348] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.845427 4213 raft_consensus.cc:374] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.845465 4213 raft_consensus.cc:729] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.845577 4213 consensus_queue.cc:260] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.845804 4213 ts_tablet_manager.cc:1398] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:07.846218 4213 tablet_bootstrap.cc:492] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45: Bootstrap starting.
I20240502 14:01:07.846297 4204 tablet_bootstrap.cc:654] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.847041 4213 tablet_bootstrap.cc:654] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.847643 4204 tablet_bootstrap.cc:492] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae: No bootstrap required, opened a new log
I20240502 14:01:07.847702 4204 ts_tablet_manager.cc:1367] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:01:07.848030 4213 tablet_bootstrap.cc:492] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45: No bootstrap required, opened a new log
I20240502 14:01:07.848062 4204 raft_consensus.cc:348] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.848114 4213 ts_tablet_manager.cc:1367] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:07.848153 4204 raft_consensus.cc:374] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.848189 4204 raft_consensus.cc:729] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.848320 4204 consensus_queue.cc:260] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.848474 4213 raft_consensus.cc:348] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.848557 4213 raft_consensus.cc:374] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.848599 4213 raft_consensus.cc:729] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.848723 4213 consensus_queue.cc:260] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.849076 4213 ts_tablet_manager.cc:1398] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:07.849084 4204 ts_tablet_manager.cc:1398] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:07.849231 4213 tablet_bootstrap.cc:492] T e6ca4998e3634ba789210e0c072e29b0 P 24fd5d6c84c1483ea8f39bb975e26e45: Bootstrap starting.
I20240502 14:01:07.849231 4204 tablet_bootstrap.cc:492] T a4d0e9d6d8e7445a9367e0f3a31c8748 P bd060dd4339f4fa5a985596bf7dec0ae: Bootstrap starting.
I20240502 14:01:07.850062 4204 tablet_bootstrap.cc:654] T a4d0e9d6d8e7445a9367e0f3a31c8748 P bd060dd4339f4fa5a985596bf7dec0ae: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.850512 4213 tablet_bootstrap.cc:654] T e6ca4998e3634ba789210e0c072e29b0 P 24fd5d6c84c1483ea8f39bb975e26e45: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.851526 4203 tablet_bootstrap.cc:492] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815: No bootstrap required, opened a new log
I20240502 14:01:07.851545 4213 tablet_bootstrap.cc:492] T e6ca4998e3634ba789210e0c072e29b0 P 24fd5d6c84c1483ea8f39bb975e26e45: No bootstrap required, opened a new log
I20240502 14:01:07.851619 4204 tablet_bootstrap.cc:492] T a4d0e9d6d8e7445a9367e0f3a31c8748 P bd060dd4339f4fa5a985596bf7dec0ae: No bootstrap required, opened a new log
I20240502 14:01:07.851619 4203 ts_tablet_manager.cc:1367] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20240502 14:01:07.851718 4204 ts_tablet_manager.cc:1367] T a4d0e9d6d8e7445a9367e0f3a31c8748 P bd060dd4339f4fa5a985596bf7dec0ae: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:01:07.851619 4213 ts_tablet_manager.cc:1367] T e6ca4998e3634ba789210e0c072e29b0 P 24fd5d6c84c1483ea8f39bb975e26e45: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:01:07.852046 4203 raft_consensus.cc:348] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.852047 4204 raft_consensus.cc:348] T a4d0e9d6d8e7445a9367e0f3a31c8748 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.852191 4204 raft_consensus.cc:374] T a4d0e9d6d8e7445a9367e0f3a31c8748 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.852155 4213 raft_consensus.cc:348] T e6ca4998e3634ba789210e0c072e29b0 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.852237 4204 raft_consensus.cc:729] T a4d0e9d6d8e7445a9367e0f3a31c8748 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.852259 4213 raft_consensus.cc:374] T e6ca4998e3634ba789210e0c072e29b0 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.852164 4203 raft_consensus.cc:374] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.852303 4213 raft_consensus.cc:729] T e6ca4998e3634ba789210e0c072e29b0 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.852332 4203 raft_consensus.cc:729] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df655d53530449fc85de33de1c229815, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.852368 4204 consensus_queue.cc:260] T a4d0e9d6d8e7445a9367e0f3a31c8748 P bd060dd4339f4fa5a985596bf7dec0ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.852406 4213 consensus_queue.cc:260] T e6ca4998e3634ba789210e0c072e29b0 P 24fd5d6c84c1483ea8f39bb975e26e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.852599 4204 ts_tablet_manager.cc:1398] T a4d0e9d6d8e7445a9367e0f3a31c8748 P bd060dd4339f4fa5a985596bf7dec0ae: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:07.852630 4213 ts_tablet_manager.cc:1398] T e6ca4998e3634ba789210e0c072e29b0 P 24fd5d6c84c1483ea8f39bb975e26e45: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:07.852780 4204 tablet_bootstrap.cc:492] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae: Bootstrap starting.
I20240502 14:01:07.852427 4203 consensus_queue.cc:260] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.853094 4203 ts_tablet_manager.cc:1398] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:01:07.852783 4213 tablet_bootstrap.cc:492] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45: Bootstrap starting.
I20240502 14:01:07.853554 4203 tablet_bootstrap.cc:492] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815: Bootstrap starting.
I20240502 14:01:07.853578 4204 tablet_bootstrap.cc:654] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.853914 4213 tablet_bootstrap.cc:654] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.854308 4203 tablet_bootstrap.cc:654] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.855530 4203 tablet_bootstrap.cc:492] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815: No bootstrap required, opened a new log
I20240502 14:01:07.855605 4203 ts_tablet_manager.cc:1367] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:01:07.855911 4203 raft_consensus.cc:348] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.855994 4203 raft_consensus.cc:374] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.856065 4206 raft_consensus.cc:482] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:07.856115 4206 raft_consensus.cc:504] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.856491 4206 leader_election.cc:288] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667), 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251)
I20240502 14:01:07.856900 4203 raft_consensus.cc:729] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df655d53530449fc85de33de1c229815, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.857156 4203 consensus_queue.cc:260] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.857432 4203 ts_tablet_manager.cc:1398] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:01:07.857985 4213 tablet_bootstrap.cc:492] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45: No bootstrap required, opened a new log
I20240502 14:01:07.858059 4213 ts_tablet_manager.cc:1367] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:01:07.858230 4204 tablet_bootstrap.cc:492] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae: No bootstrap required, opened a new log
I20240502 14:01:07.858292 4204 ts_tablet_manager.cc:1367] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240502 14:01:07.858428 4213 raft_consensus.cc:348] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.858505 4213 raft_consensus.cc:374] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.858534 4213 raft_consensus.cc:729] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.858628 4213 consensus_queue.cc:260] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.858841 4213 ts_tablet_manager.cc:1398] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:07.859016 4213 tablet_bootstrap.cc:492] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45: Bootstrap starting.
I20240502 14:01:07.859246 4204 raft_consensus.cc:348] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.859391 4204 raft_consensus.cc:374] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.859436 4204 raft_consensus.cc:729] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.859591 4204 consensus_queue.cc:260] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.859804 4213 tablet_bootstrap.cc:654] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.860898 4204 ts_tablet_manager.cc:1398] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:01:07.861451 4006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ac79a5b1f514bdcb36e0db4ff6e3a59" candidate_uuid: "df655d53530449fc85de33de1c229815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" is_pre_election: true
I20240502 14:01:07.861570 4006 raft_consensus.cc:2454] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df655d53530449fc85de33de1c229815 in term 0.
I20240502 14:01:07.861860 4122 leader_election.cc:302] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd060dd4339f4fa5a985596bf7dec0ae, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:07.862015 4206 raft_consensus.cc:2781] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:07.862059 4206 raft_consensus.cc:482] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:07.862094 4206 raft_consensus.cc:3037] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:07.862973 4206 raft_consensus.cc:504] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.863270 4206 leader_election.cc:288] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [CANDIDATE]: Term 1 election: Requested vote from peers bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667), 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251)
I20240502 14:01:07.863773 4204 tablet_bootstrap.cc:492] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae: Bootstrap starting.
I20240502 14:01:07.863888 4006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ac79a5b1f514bdcb36e0db4ff6e3a59" candidate_uuid: "df655d53530449fc85de33de1c229815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
I20240502 14:01:07.863982 4006 raft_consensus.cc:3037] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:07.864578 4204 tablet_bootstrap.cc:654] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.864820 4006 raft_consensus.cc:2454] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df655d53530449fc85de33de1c229815 in term 1.
I20240502 14:01:07.865146 4122 leader_election.cc:302] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd060dd4339f4fa5a985596bf7dec0ae, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:07.865171 4082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ac79a5b1f514bdcb36e0db4ff6e3a59" candidate_uuid: "df655d53530449fc85de33de1c229815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" is_pre_election: true
I20240502 14:01:07.865280 4082 raft_consensus.cc:2454] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df655d53530449fc85de33de1c229815 in term 0.
I20240502 14:01:07.865419 4206 raft_consensus.cc:2781] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:07.865521 4081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ac79a5b1f514bdcb36e0db4ff6e3a59" candidate_uuid: "df655d53530449fc85de33de1c229815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
I20240502 14:01:07.865609 4081 raft_consensus.cc:3037] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:07.866171 4204 tablet_bootstrap.cc:492] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae: No bootstrap required, opened a new log
I20240502 14:01:07.866251 4204 ts_tablet_manager.cc:1367] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:01:07.866537 4081 raft_consensus.cc:2454] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df655d53530449fc85de33de1c229815 in term 1.
I20240502 14:01:07.866600 4213 tablet_bootstrap.cc:492] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45: No bootstrap required, opened a new log
I20240502 14:01:07.866669 4213 ts_tablet_manager.cc:1367] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240502 14:01:07.866633 4204 raft_consensus.cc:348] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.866709 4204 raft_consensus.cc:374] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.866739 4204 raft_consensus.cc:729] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.866880 4204 consensus_queue.cc:260] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.867056 4213 raft_consensus.cc:348] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.867136 4213 raft_consensus.cc:374] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.867172 4213 raft_consensus.cc:729] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.865610 4206 raft_consensus.cc:686] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [term 1 LEADER]: Becoming Leader. State: Replica: df655d53530449fc85de33de1c229815, State: Running, Role: LEADER
I20240502 14:01:07.867264 4213 consensus_queue.cc:260] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.867364 4206 consensus_queue.cc:237] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.867491 4213 ts_tablet_manager.cc:1398] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:07.867655 4213 tablet_bootstrap.cc:492] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45: Bootstrap starting.
I20240502 14:01:07.868398 4204 ts_tablet_manager.cc:1398] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:07.868453 4213 tablet_bootstrap.cc:654] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.869009 3902 catalog_manager.cc:5484] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 reported cstate change: term changed from 0 to 1, leader changed from <none> to df655d53530449fc85de33de1c229815 (127.0.145.67). New cstate: current_term: 1 leader_uuid: "df655d53530449fc85de33de1c229815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: HEALTHY } } }
I20240502 14:01:07.869179 4204 tablet_bootstrap.cc:492] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae: Bootstrap starting.
I20240502 14:01:07.869572 4217 raft_consensus.cc:482] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:07.869657 4217 raft_consensus.cc:504] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.870016 4204 tablet_bootstrap.cc:654] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.870083 4217 leader_election.cc:288] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:07.872848 4204 tablet_bootstrap.cc:492] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae: No bootstrap required, opened a new log
I20240502 14:01:07.872943 4204 ts_tablet_manager.cc:1367] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:01:07.873632 4204 raft_consensus.cc:348] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.873720 4204 raft_consensus.cc:374] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.873749 4204 raft_consensus.cc:729] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.873857 4204 consensus_queue.cc:260] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.874090 4204 ts_tablet_manager.cc:1398] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:07.874244 4204 tablet_bootstrap.cc:492] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae: Bootstrap starting.
I20240502 14:01:07.875039 4204 tablet_bootstrap.cc:654] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.876449 4213 tablet_bootstrap.cc:492] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45: No bootstrap required, opened a new log
I20240502 14:01:07.876533 4213 ts_tablet_manager.cc:1367] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 14:01:07.876724 4204 tablet_bootstrap.cc:492] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae: No bootstrap required, opened a new log
I20240502 14:01:07.876860 4204 ts_tablet_manager.cc:1367] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:01:07.876936 4213 raft_consensus.cc:348] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.877018 4213 raft_consensus.cc:374] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.877046 4213 raft_consensus.cc:729] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.877239 4157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba8942501c440b18f71c7106878d0c4" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df655d53530449fc85de33de1c229815" is_pre_election: true
I20240502 14:01:07.877363 4157 raft_consensus.cc:2454] T dba8942501c440b18f71c7106878d0c4 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 0.
I20240502 14:01:07.877640 4048 leader_election.cc:302] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:07.877794 4204 raft_consensus.cc:348] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.877805 4217 raft_consensus.cc:2781] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:07.877972 4217 raft_consensus.cc:482] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:07.878015 4217 raft_consensus.cc:3037] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:07.878741 4006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba8942501c440b18f71c7106878d0c4" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" is_pre_election: true
I20240502 14:01:07.878810 4217 raft_consensus.cc:504] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.879143 4217 leader_election.cc:288] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 1 election: Requested vote from peers bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:07.877848 4213 consensus_queue.cc:260] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.879369 4220 raft_consensus.cc:482] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:07.877938 4204 raft_consensus.cc:374] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.879473 4220 raft_consensus.cc:504] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.879535 4204 raft_consensus.cc:729] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.879650 4204 consensus_queue.cc:260] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.879858 4220 leader_election.cc:288] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:07.879995 4005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba8942501c440b18f71c7106878d0c4" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
I20240502 14:01:07.880327 4006 raft_consensus.cc:2454] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 0.
I20240502 14:01:07.881227 4217 raft_consensus.cc:482] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:07.881305 4217 raft_consensus.cc:504] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.881410 4157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba8942501c440b18f71c7106878d0c4" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df655d53530449fc85de33de1c229815"
I20240502 14:01:07.881496 4157 raft_consensus.cc:3037] T dba8942501c440b18f71c7106878d0c4 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:07.881608 4217 leader_election.cc:288] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:07.882031 4156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4d0e9d6d8e7445a9367e0f3a31c8748" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df655d53530449fc85de33de1c229815" is_pre_election: true
I20240502 14:01:07.882144 4156 raft_consensus.cc:2454] T a4d0e9d6d8e7445a9367e0f3a31c8748 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 0.
I20240502 14:01:07.882323 4157 raft_consensus.cc:2454] T dba8942501c440b18f71c7106878d0c4 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 1.
I20240502 14:01:07.882367 4048 leader_election.cc:302] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:07.879408 4213 ts_tablet_manager.cc:1398] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:01:07.882537 4048 leader_election.cc:302] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, df655d53530449fc85de33de1c229815; no voters: bd060dd4339f4fa5a985596bf7dec0ae
I20240502 14:01:07.882583 4204 ts_tablet_manager.cc:1398] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 14:01:07.882733 4219 raft_consensus.cc:2781] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:07.882790 4219 raft_consensus.cc:482] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:07.882850 4219 raft_consensus.cc:3037] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:07.883105 4213 tablet_bootstrap.cc:492] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45: Bootstrap starting.
I20240502 14:01:07.883638 4219 raft_consensus.cc:504] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.883945 4213 tablet_bootstrap.cc:654] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:07.884227 4005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4d0e9d6d8e7445a9367e0f3a31c8748" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
I20240502 14:01:07.884315 4005 raft_consensus.cc:3037] T a4d0e9d6d8e7445a9367e0f3a31c8748 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:07.883953 4219 leader_election.cc:288] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 1 election: Requested vote from peers bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:07.884914 4219 raft_consensus.cc:2781] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:07.884970 4219 raft_consensus.cc:686] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Becoming Leader. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Running, Role: LEADER
I20240502 14:01:07.885080 4219 consensus_queue.cc:237] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.885114 4005 raft_consensus.cc:2454] T a4d0e9d6d8e7445a9367e0f3a31c8748 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 1.
I20240502 14:01:07.885381 4047 leader_election.cc:302] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, bd060dd4339f4fa5a985596bf7dec0ae; no voters:
I20240502 14:01:07.885653 4219 raft_consensus.cc:2781] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:07.885735 4219 raft_consensus.cc:686] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Becoming Leader. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Running, Role: LEADER
I20240502 14:01:07.885844 4219 consensus_queue.cc:237] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.885864 4157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4d0e9d6d8e7445a9367e0f3a31c8748" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df655d53530449fc85de33de1c229815"
I20240502 14:01:07.885973 4157 raft_consensus.cc:3037] T a4d0e9d6d8e7445a9367e0f3a31c8748 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:07.885965 4081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7908edf8bbe8419ab21d0c93cd2a3b35" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" is_pre_election: true
I20240502 14:01:07.886135 4081 raft_consensus.cc:2454] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 0.
I20240502 14:01:07.886682 4157 raft_consensus.cc:2454] T a4d0e9d6d8e7445a9367e0f3a31c8748 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 1.
I20240502 14:01:07.887022 4213 tablet_bootstrap.cc:492] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45: No bootstrap required, opened a new log
I20240502 14:01:07.887015 3972 leader_election.cc:302] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, bd060dd4339f4fa5a985596bf7dec0ae; no voters:
I20240502 14:01:07.887178 4213 ts_tablet_manager.cc:1367] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:01:07.887315 4220 raft_consensus.cc:2781] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:07.887384 4220 raft_consensus.cc:482] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:07.887431 4220 raft_consensus.cc:3037] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:07.887557 3903 catalog_manager.cc:5484] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66). New cstate: current_term: 1 leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:07.887609 4213 raft_consensus.cc:348] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.887717 4213 raft_consensus.cc:374] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:07.887753 4213 raft_consensus.cc:729] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Initialized, Role: FOLLOWER
I20240502 14:01:07.887861 4213 consensus_queue.cc:260] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.888108 4213 ts_tablet_manager.cc:1398] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:07.888188 4220 raft_consensus.cc:504] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.888481 4220 leader_election.cc:288] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 1 election: Requested vote from peers 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:07.885964 4006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4d0e9d6d8e7445a9367e0f3a31c8748" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" is_pre_election: true
I20240502 14:01:07.888954 4081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7908edf8bbe8419ab21d0c93cd2a3b35" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
I20240502 14:01:07.888984 4006 raft_consensus.cc:2362] T a4d0e9d6d8e7445a9367e0f3a31c8748 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 1. Re-sending same reply.
I20240502 14:01:07.889024 4081 raft_consensus.cc:3037] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:07.889264 4157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7908edf8bbe8419ab21d0c93cd2a3b35" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df655d53530449fc85de33de1c229815" is_pre_election: true
I20240502 14:01:07.889317 4156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7908edf8bbe8419ab21d0c93cd2a3b35" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df655d53530449fc85de33de1c229815"
I20240502 14:01:07.889360 4157 raft_consensus.cc:2454] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 0.
I20240502 14:01:07.889693 4081 raft_consensus.cc:2454] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 1.
I20240502 14:01:07.890167 3972 leader_election.cc:302] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, bd060dd4339f4fa5a985596bf7dec0ae; no voters: df655d53530449fc85de33de1c229815
I20240502 14:01:07.890314 4220 raft_consensus.cc:2781] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:07.890412 4220 raft_consensus.cc:686] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 LEADER]: Becoming Leader. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Running, Role: LEADER
I20240502 14:01:07.890517 4220 consensus_queue.cc:237] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.891291 3903 catalog_manager.cc:5484] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66). New cstate: current_term: 1 leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:07.891892 3902 catalog_manager.cc:5484] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae reported cstate change: term changed from 0 to 1, leader changed from <none> to bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65). New cstate: current_term: 1 leader_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:07.923694 4223 raft_consensus.cc:482] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:07.923797 4223 raft_consensus.cc:504] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } }
I20240502 14:01:07.924139 4223 leader_election.cc:288] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df655d53530449fc85de33de1c229815 (127.0.145.67:34309), 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251)
I20240502 14:01:07.924423 4157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb4c3dcde8764e17bb084cb37fef706f" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df655d53530449fc85de33de1c229815" is_pre_election: true
I20240502 14:01:07.924499 4081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb4c3dcde8764e17bb084cb37fef706f" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" is_pre_election: true
I20240502 14:01:07.924535 4157 raft_consensus.cc:2454] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 0.
I20240502 14:01:07.924620 4081 raft_consensus.cc:2454] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 0.
I20240502 14:01:07.924820 3972 leader_election.cc:302] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd060dd4339f4fa5a985596bf7dec0ae, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:07.924978 4223 raft_consensus.cc:2781] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:07.925036 4223 raft_consensus.cc:482] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:07.925082 4223 raft_consensus.cc:3037] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:07.925802 4223 raft_consensus.cc:504] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } }
I20240502 14:01:07.926064 4223 leader_election.cc:288] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 1 election: Requested vote from peers df655d53530449fc85de33de1c229815 (127.0.145.67:34309), 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251)
I20240502 14:01:07.926311 4157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb4c3dcde8764e17bb084cb37fef706f" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df655d53530449fc85de33de1c229815"
I20240502 14:01:07.926396 4157 raft_consensus.cc:3037] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:07.926499 4081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb4c3dcde8764e17bb084cb37fef706f" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
I20240502 14:01:07.926605 4081 raft_consensus.cc:3037] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:07.927129 4157 raft_consensus.cc:2454] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 1.
I20240502 14:01:07.927295 4081 raft_consensus.cc:2454] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 1.
I20240502 14:01:07.927383 3972 leader_election.cc:302] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd060dd4339f4fa5a985596bf7dec0ae, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:07.927524 4223 raft_consensus.cc:2781] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:07.927629 4223 raft_consensus.cc:686] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [term 1 LEADER]: Becoming Leader. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Running, Role: LEADER
I20240502 14:01:07.927747 4223 consensus_queue.cc:237] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } }
I20240502 14:01:07.929159 3902 catalog_manager.cc:5484] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae reported cstate change: term changed from 0 to 1, leader changed from <none> to bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65). New cstate: current_term: 1 leader_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } }
I20240502 14:01:07.935801 4255 raft_consensus.cc:482] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:07.935894 4255 raft_consensus.cc:504] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.936143 4255 leader_election.cc:288] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:07.936383 4006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c965f0e11bc1433f8ccdc20c98b4b4b9" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" is_pre_election: true
I20240502 14:01:07.936456 4157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c965f0e11bc1433f8ccdc20c98b4b4b9" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df655d53530449fc85de33de1c229815" is_pre_election: true
I20240502 14:01:07.936496 4006 raft_consensus.cc:2454] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 0.
I20240502 14:01:07.936548 4157 raft_consensus.cc:2454] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 0.
I20240502 14:01:07.936784 4048 leader_election.cc:302] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:07.936937 4255 raft_consensus.cc:2781] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:07.936995 4255 raft_consensus.cc:482] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:07.937037 4255 raft_consensus.cc:3037] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:07.938012 4255 raft_consensus.cc:504] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.938262 4255 leader_election.cc:288] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 1 election: Requested vote from peers bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:07.938516 4157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c965f0e11bc1433f8ccdc20c98b4b4b9" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df655d53530449fc85de33de1c229815"
I20240502 14:01:07.938520 4006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c965f0e11bc1433f8ccdc20c98b4b4b9" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
I20240502 14:01:07.938607 4157 raft_consensus.cc:3037] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:07.938622 4006 raft_consensus.cc:3037] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:07.939182 4157 raft_consensus.cc:2454] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 1.
I20240502 14:01:07.939232 4006 raft_consensus.cc:2454] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 1.
I20240502 14:01:07.939419 4048 leader_election.cc:302] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:07.939594 4255 raft_consensus.cc:2781] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:07.939666 4255 raft_consensus.cc:686] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Becoming Leader. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Running, Role: LEADER
I20240502 14:01:07.939777 4255 consensus_queue.cc:237] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:07.941071 3902 catalog_manager.cc:5484] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66). New cstate: current_term: 1 leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:08.016360 4223 raft_consensus.cc:482] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:08.016481 4223 raft_consensus.cc:504] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:08.016876 4223 leader_election.cc:288] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:08.017194 4081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975becfb8df0430fa8c748a58dcbcfb1" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" is_pre_election: true
I20240502 14:01:08.017194 4157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975becfb8df0430fa8c748a58dcbcfb1" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df655d53530449fc85de33de1c229815" is_pre_election: true
I20240502 14:01:08.017359 4157 raft_consensus.cc:2454] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 0.
I20240502 14:01:08.017359 4081 raft_consensus.cc:2454] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 0.
I20240502 14:01:08.017589 3972 leader_election.cc:302] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd060dd4339f4fa5a985596bf7dec0ae, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:08.017720 4223 raft_consensus.cc:2781] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:08.017778 4223 raft_consensus.cc:482] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:08.017802 4223 raft_consensus.cc:3037] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:08.018553 4223 raft_consensus.cc:504] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:08.018852 4223 leader_election.cc:288] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 1 election: Requested vote from peers 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:08.019102 4081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975becfb8df0430fa8c748a58dcbcfb1" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
I20240502 14:01:08.019160 4157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975becfb8df0430fa8c748a58dcbcfb1" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df655d53530449fc85de33de1c229815"
I20240502 14:01:08.019232 4081 raft_consensus.cc:3037] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:08.019263 4157 raft_consensus.cc:3037] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:08.019865 4157 raft_consensus.cc:2454] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 1.
I20240502 14:01:08.019906 4081 raft_consensus.cc:2454] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 1.
I20240502 14:01:08.020110 3972 leader_election.cc:302] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, bd060dd4339f4fa5a985596bf7dec0ae; no voters:
I20240502 14:01:08.020263 4223 raft_consensus.cc:2781] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:08.020352 4223 raft_consensus.cc:686] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 LEADER]: Becoming Leader. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Running, Role: LEADER
I20240502 14:01:08.020485 4223 consensus_queue.cc:237] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:08.021890 3902 catalog_manager.cc:5484] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae reported cstate change: term changed from 0 to 1, leader changed from <none> to bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65). New cstate: current_term: 1 leader_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:08.033507 4223 raft_consensus.cc:482] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:08.033587 4223 raft_consensus.cc:504] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:08.033839 4223 leader_election.cc:288] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:08.034085 4081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6ca4998e3634ba789210e0c072e29b0" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" is_pre_election: true
I20240502 14:01:08.034168 4157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6ca4998e3634ba789210e0c072e29b0" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df655d53530449fc85de33de1c229815" is_pre_election: true
I20240502 14:01:08.034174 4081 raft_consensus.cc:2454] T e6ca4998e3634ba789210e0c072e29b0 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 0.
I20240502 14:01:08.034282 4157 raft_consensus.cc:2454] T e6ca4998e3634ba789210e0c072e29b0 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 0.
I20240502 14:01:08.034436 3972 leader_election.cc:302] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, bd060dd4339f4fa5a985596bf7dec0ae; no voters:
I20240502 14:01:08.034585 4223 raft_consensus.cc:2781] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:08.034631 4223 raft_consensus.cc:482] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:08.034673 4223 raft_consensus.cc:3037] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:08.035293 4223 raft_consensus.cc:504] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:08.035570 4223 leader_election.cc:288] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 1 election: Requested vote from peers 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:08.035792 4081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6ca4998e3634ba789210e0c072e29b0" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
I20240502 14:01:08.035811 4157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6ca4998e3634ba789210e0c072e29b0" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df655d53530449fc85de33de1c229815"
I20240502 14:01:08.035923 4081 raft_consensus.cc:3037] T e6ca4998e3634ba789210e0c072e29b0 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:08.035930 4157 raft_consensus.cc:3037] T e6ca4998e3634ba789210e0c072e29b0 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:08.036706 4157 raft_consensus.cc:2454] T e6ca4998e3634ba789210e0c072e29b0 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 1.
I20240502 14:01:08.036706 4081 raft_consensus.cc:2454] T e6ca4998e3634ba789210e0c072e29b0 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 1.
I20240502 14:01:08.037003 3972 leader_election.cc:302] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd060dd4339f4fa5a985596bf7dec0ae, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:08.037154 4223 raft_consensus.cc:2781] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:08.037237 4223 raft_consensus.cc:686] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 LEADER]: Becoming Leader. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Running, Role: LEADER
I20240502 14:01:08.037356 4223 consensus_queue.cc:237] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:08.038681 3902 catalog_manager.cc:5484] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae reported cstate change: term changed from 0 to 1, leader changed from <none> to bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65). New cstate: current_term: 1 leader_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:08.046185 4223 raft_consensus.cc:482] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:08.046260 4223 raft_consensus.cc:504] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:08.046519 4223 leader_election.cc:288] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:08.046753 4081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5172e93df3a4fb5a70188b0507639e6" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" is_pre_election: true
I20240502 14:01:08.046788 4157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5172e93df3a4fb5a70188b0507639e6" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df655d53530449fc85de33de1c229815" is_pre_election: true
I20240502 14:01:08.046877 4157 raft_consensus.cc:2454] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 0.
I20240502 14:01:08.046877 4081 raft_consensus.cc:2454] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 0.
I20240502 14:01:08.047096 3972 leader_election.cc:302] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd060dd4339f4fa5a985596bf7dec0ae, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:08.047232 4223 raft_consensus.cc:2781] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:08.047287 4223 raft_consensus.cc:482] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:08.047331 4223 raft_consensus.cc:3037] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:08.047937 4223 raft_consensus.cc:504] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:08.048184 4223 leader_election.cc:288] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 1 election: Requested vote from peers 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:08.048447 4081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5172e93df3a4fb5a70188b0507639e6" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
I20240502 14:01:08.048494 4157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5172e93df3a4fb5a70188b0507639e6" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df655d53530449fc85de33de1c229815"
I20240502 14:01:08.048547 4081 raft_consensus.cc:3037] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:08.048556 4157 raft_consensus.cc:3037] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:08.049340 4081 raft_consensus.cc:2454] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 1.
I20240502 14:01:08.049340 4157 raft_consensus.cc:2454] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 1.
I20240502 14:01:08.049599 3972 leader_election.cc:302] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd060dd4339f4fa5a985596bf7dec0ae, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:08.049727 4223 raft_consensus.cc:2781] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:08.049808 4223 raft_consensus.cc:686] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 LEADER]: Becoming Leader. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Running, Role: LEADER
I20240502 14:01:08.049919 4223 consensus_queue.cc:237] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:08.050721 4206 raft_consensus.cc:482] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:08.050797 4206 raft_consensus.cc:504] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:08.051074 4206 leader_election.cc:288] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667), 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251)
I20240502 14:01:08.051263 3902 catalog_manager.cc:5484] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae reported cstate change: term changed from 0 to 1, leader changed from <none> to bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65). New cstate: current_term: 1 leader_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:08.051373 4006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144739b71bf6474d90399816120b3b59" candidate_uuid: "df655d53530449fc85de33de1c229815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" is_pre_election: true
I20240502 14:01:08.051466 4006 raft_consensus.cc:2454] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df655d53530449fc85de33de1c229815 in term 0.
I20240502 14:01:08.051636 4081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144739b71bf6474d90399816120b3b59" candidate_uuid: "df655d53530449fc85de33de1c229815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" is_pre_election: true
I20240502 14:01:08.051679 4122 leader_election.cc:302] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd060dd4339f4fa5a985596bf7dec0ae, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:08.051729 4081 raft_consensus.cc:2454] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df655d53530449fc85de33de1c229815 in term 0.
I20240502 14:01:08.051833 4206 raft_consensus.cc:2781] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:08.051879 4206 raft_consensus.cc:482] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:08.051934 4206 raft_consensus.cc:3037] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:08.052582 4206 raft_consensus.cc:504] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:08.052882 4206 leader_election.cc:288] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [CANDIDATE]: Term 1 election: Requested vote from peers bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667), 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251)
I20240502 14:01:08.053181 4081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144739b71bf6474d90399816120b3b59" candidate_uuid: "df655d53530449fc85de33de1c229815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
I20240502 14:01:08.053279 4081 raft_consensus.cc:3037] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:08.053293 4006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144739b71bf6474d90399816120b3b59" candidate_uuid: "df655d53530449fc85de33de1c229815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
I20240502 14:01:08.053388 4006 raft_consensus.cc:3037] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:08.053876 4081 raft_consensus.cc:2454] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df655d53530449fc85de33de1c229815 in term 1.
I20240502 14:01:08.053982 4006 raft_consensus.cc:2454] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df655d53530449fc85de33de1c229815 in term 1.
I20240502 14:01:08.054126 4123 leader_election.cc:302] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:08.054318 4206 raft_consensus.cc:2781] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:08.054383 4206 raft_consensus.cc:686] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 1 LEADER]: Becoming Leader. State: Replica: df655d53530449fc85de33de1c229815, State: Running, Role: LEADER
I20240502 14:01:08.054476 4206 consensus_queue.cc:237] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:08.055752 3902 catalog_manager.cc:5484] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 reported cstate change: term changed from 0 to 1, leader changed from <none> to df655d53530449fc85de33de1c229815 (127.0.145.67). New cstate: current_term: 1 leader_uuid: "df655d53530449fc85de33de1c229815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: HEALTHY } } }
I20240502 14:01:08.065814 4255 raft_consensus.cc:482] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:08.065887 4255 raft_consensus.cc:504] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:08.066181 4255 leader_election.cc:288] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:08.066437 4006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d0d4aeaec264d04bae2c34e6edc6ec7" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" is_pre_election: true
I20240502 14:01:08.066476 4157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d0d4aeaec264d04bae2c34e6edc6ec7" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df655d53530449fc85de33de1c229815" is_pre_election: true
I20240502 14:01:08.066536 4006 raft_consensus.cc:2454] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 0.
I20240502 14:01:08.066557 4157 raft_consensus.cc:2454] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 0.
I20240502 14:01:08.066746 4047 leader_election.cc:302] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, bd060dd4339f4fa5a985596bf7dec0ae; no voters:
I20240502 14:01:08.066918 4255 raft_consensus.cc:2781] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:08.066975 4255 raft_consensus.cc:482] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:08.067010 4255 raft_consensus.cc:3037] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:08.067682 4255 raft_consensus.cc:504] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:08.067965 4255 leader_election.cc:288] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 1 election: Requested vote from peers bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:08.068223 4006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d0d4aeaec264d04bae2c34e6edc6ec7" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
I20240502 14:01:08.068223 4157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d0d4aeaec264d04bae2c34e6edc6ec7" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df655d53530449fc85de33de1c229815"
I20240502 14:01:08.068351 4006 raft_consensus.cc:3037] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:08.068351 4157 raft_consensus.cc:3037] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:08.069136 4006 raft_consensus.cc:2454] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 1.
I20240502 14:01:08.069136 4157 raft_consensus.cc:2454] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 1.
I20240502 14:01:08.069432 4048 leader_election.cc:302] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, bd060dd4339f4fa5a985596bf7dec0ae, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:08.069618 4255 raft_consensus.cc:2781] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:08.069697 4255 raft_consensus.cc:686] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Becoming Leader. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Running, Role: LEADER
I20240502 14:01:08.069809 4255 consensus_queue.cc:237] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:08.071112 3902 catalog_manager.cc:5484] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66). New cstate: current_term: 1 leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:08.171372 4255 raft_consensus.cc:482] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:08.171473 4255 raft_consensus.cc:504] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:08.171819 4255 leader_election.cc:288] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:08.172111 4006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46b4a50dee854f62aac3a02889525b40" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" is_pre_election: true
I20240502 14:01:08.172111 4157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46b4a50dee854f62aac3a02889525b40" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df655d53530449fc85de33de1c229815" is_pre_election: true
I20240502 14:01:08.172222 4006 raft_consensus.cc:2454] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 0.
I20240502 14:01:08.172261 4157 raft_consensus.cc:2454] T 46b4a50dee854f62aac3a02889525b40 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 0.
I20240502 14:01:08.172456 4047 leader_election.cc:302] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, bd060dd4339f4fa5a985596bf7dec0ae; no voters:
I20240502 14:01:08.172605 4255 raft_consensus.cc:2781] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:08.172653 4255 raft_consensus.cc:482] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:08.172677 4255 raft_consensus.cc:3037] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:08.173374 4255 raft_consensus.cc:504] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:08.173653 4255 leader_election.cc:288] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 1 election: Requested vote from peers bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:08.173864 4006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46b4a50dee854f62aac3a02889525b40" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
I20240502 14:01:08.173923 4157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46b4a50dee854f62aac3a02889525b40" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df655d53530449fc85de33de1c229815"
I20240502 14:01:08.173964 4006 raft_consensus.cc:3037] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:08.174021 4157 raft_consensus.cc:3037] T 46b4a50dee854f62aac3a02889525b40 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:08.174590 4006 raft_consensus.cc:2454] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 1.
I20240502 14:01:08.174610 4157 raft_consensus.cc:2454] T 46b4a50dee854f62aac3a02889525b40 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 1.
I20240502 14:01:08.174816 4048 leader_election.cc:302] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:08.174964 4255 raft_consensus.cc:2781] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:08.175029 4255 raft_consensus.cc:686] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Becoming Leader. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Running, Role: LEADER
I20240502 14:01:08.175144 4255 consensus_queue.cc:237] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:08.176396 3902 catalog_manager.cc:5484] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66). New cstate: current_term: 1 leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:08.264838 4157 raft_consensus.cc:3037] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:08.264938 4081 raft_consensus.cc:1261] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:08.265900 4157 raft_consensus.cc:1261] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:08.266498 4220 consensus_queue.cc:1035] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:08.268153 4157 raft_consensus.cc:1261] T e6ca4998e3634ba789210e0c072e29b0 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:08.270959 4156 raft_consensus.cc:1261] T a4d0e9d6d8e7445a9367e0f3a31c8748 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:08.272367 4217 consensus_queue.cc:1035] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:08.272797 4156 raft_consensus.cc:1261] T dba8942501c440b18f71c7106878d0c4 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:08.273353 4155 raft_consensus.cc:1261] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:08.273661 4082 raft_consensus.cc:1261] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:08.273922 4006 raft_consensus.cc:1261] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:08.274314 4005 raft_consensus.cc:3037] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:08.274813 4217 consensus_queue.cc:1035] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:08.275491 4217 consensus_queue.cc:1035] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:08.273681 4255 consensus_queue.cc:1035] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:01:08.279510 4003 raft_consensus.cc:1261] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:01:08.279778 4156 raft_consensus.cc:1261] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:08.279840 4255 consensus_queue.cc:1035] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:08.280025 4081 raft_consensus.cc:1261] T e6ca4998e3634ba789210e0c072e29b0 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:08.280171 4220 consensus_queue.cc:1035] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:08.275331 4223 consensus_queue.cc:1035] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:08.281922 4157 raft_consensus.cc:1261] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:08.282191 4282 consensus_queue.cc:1035] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20240502 14:01:08.283140 4255 consensus_queue.cc:1035] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:01:08.284224 4220 consensus_queue.cc:1035] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:08.283306 4004 raft_consensus.cc:1261] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:08.285327 4155 raft_consensus.cc:1261] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:08.282548 4283 consensus_queue.cc:1035] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20240502 14:01:08.284286 4153 raft_consensus.cc:1261] T 46b4a50dee854f62aac3a02889525b40 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:01:08.287042 4080 raft_consensus.cc:1261] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:08.286414 4003 raft_consensus.cc:1261] T a4d0e9d6d8e7445a9367e0f3a31c8748 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:08.288115 4004 raft_consensus.cc:1261] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:08.290153 4002 raft_consensus.cc:1261] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:08.290727 4282 consensus_queue.cc:1035] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240502 14:01:08.290808 4005 raft_consensus.cc:1261] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:08.290926 4217 consensus_queue.cc:1035] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240502 14:01:08.293731 4217 consensus_queue.cc:1035] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:08.290619 4283 consensus_queue.cc:1035] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20240502 14:01:08.294195 4078 raft_consensus.cc:1261] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:08.294452 4151 raft_consensus.cc:1261] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:08.294605 4079 raft_consensus.cc:1261] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:08.294917 4077 raft_consensus.cc:1261] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:08.295248 4245 consensus_queue.cc:1035] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240502 14:01:08.295411 4206 consensus_queue.cc:1035] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:08.291003 4255 consensus_queue.cc:1035] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20240502 14:01:08.296360 4220 consensus_queue.cc:1035] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:08.296604 4282 consensus_queue.cc:1035] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:08.296890 4206 consensus_queue.cc:1035] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:08.297554 4291 consensus_queue.cc:1035] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20240502 14:01:08.305162 4301 consensus_queue.cc:1035] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20240502 14:01:08.321586 4285 mvcc.cc:205] Tried to move back new op lower bound from 7023241086020997120 to 7023241084461752320. Current Snapshot: MvccSnapshot[applied={T|T < 7023241086020997120}]
I20240502 14:01:08.327854 4303 mvcc.cc:205] Tried to move back new op lower bound from 7023241086116110336 to 7023241085133660160. Current Snapshot: MvccSnapshot[applied={T|T < 7023241086059008000}]
I20240502 14:01:08.353792 4279 mvcc.cc:205] Tried to move back new op lower bound from 7023241086061256704 to 7023241085646532608. Current Snapshot: MvccSnapshot[applied={T|T < 7023241086061256704 or (T in {7023241086061256704})}]
I20240502 14:01:08.629449 3957 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:01:08.650292 4005 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7908edf8bbe8419ab21d0c93cd2a3b35"
dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
mode: GRACEFUL
new_leader_uuid: "df655d53530449fc85de33de1c229815"
from {username='slave'} at 127.0.0.1:57080
I20240502 14:01:08.650449 4005 raft_consensus.cc:595] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 LEADER]: Received request to transfer leadership to TS df655d53530449fc85de33de1c229815
I20240502 14:01:08.660895 4082 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c965f0e11bc1433f8ccdc20c98b4b4b9"
dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
mode: GRACEFUL
new_leader_uuid: "df655d53530449fc85de33de1c229815"
from {username='slave'} at 127.0.0.1:59812
I20240502 14:01:08.661028 4082 raft_consensus.cc:595] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Received request to transfer leadership to TS df655d53530449fc85de33de1c229815
W20240502 14:01:08.661336 3977 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57076: Service unavailable: leader transfer in progress
I20240502 14:01:08.661821 3957 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240502 14:01:08.661933 3957 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:01:08.662626 4339 raft_consensus.cc:979] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae: : Instructing follower df655d53530449fc85de33de1c229815 to start an election
I20240502 14:01:08.662704 4339 raft_consensus.cc:1067] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 LEADER]: Signalling peer df655d53530449fc85de33de1c229815 to start an election
I20240502 14:01:08.668673 4148 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7908edf8bbe8419ab21d0c93cd2a3b35"
dest_uuid: "df655d53530449fc85de33de1c229815"
from {username='slave'} at 127.0.0.1:33594
I20240502 14:01:08.668812 4148 raft_consensus.cc:482] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:08.668907 4148 raft_consensus.cc:3037] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:08.670177 3977 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57076: Service unavailable: leader transfer in progress
I20240502 14:01:08.670904 4148 raft_consensus.cc:504] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:08.671078 4154 raft_consensus.cc:1226] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [term 2 FOLLOWER]: Rejecting Update request from peer bd060dd4339f4fa5a985596bf7dec0ae for earlier term 1. Current term is 2. Ops: []
I20240502 14:01:08.673835 4148 leader_election.cc:288] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [CANDIDATE]: Term 2 election: Requested vote from peers bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667), 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251)
I20240502 14:01:08.674108 4005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7908edf8bbe8419ab21d0c93cd2a3b35" candidate_uuid: "df655d53530449fc85de33de1c229815" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: true dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
I20240502 14:01:08.674212 4005 raft_consensus.cc:3032] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:01:08.674259 4005 raft_consensus.cc:729] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Running, Role: LEADER
I20240502 14:01:08.674450 4005 consensus_queue.cc:260] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 1.22, Last appended by leader: 22, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:08.674628 4005 raft_consensus.cc:3037] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:08.675215 3977 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57076: Illegal state: replica bd060dd4339f4fa5a985596bf7dec0ae is not leader of this config: current role FOLLOWER
I20240502 14:01:08.675504 4005 raft_consensus.cc:2454] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df655d53530449fc85de33de1c229815 in term 2.
I20240502 14:01:08.675736 4122 leader_election.cc:302] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd060dd4339f4fa5a985596bf7dec0ae, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:08.675899 4245 raft_consensus.cc:2781] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:08.675977 4245 raft_consensus.cc:686] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [term 2 LEADER]: Becoming Leader. State: Replica: df655d53530449fc85de33de1c229815, State: Running, Role: LEADER
I20240502 14:01:08.676091 4245 consensus_queue.cc:237] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 1.22, Last appended by leader: 22, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:08.674433 4077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7908edf8bbe8419ab21d0c93cd2a3b35" candidate_uuid: "df655d53530449fc85de33de1c229815" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: true dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
I20240502 14:01:08.677306 4077 raft_consensus.cc:3037] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:08.678001 3903 catalog_manager.cc:5484] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 reported cstate change: term changed from 1 to 2, leader changed from bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65) to df655d53530449fc85de33de1c229815 (127.0.145.67). New cstate: current_term: 2 leader_uuid: "df655d53530449fc85de33de1c229815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: HEALTHY } } }
I20240502 14:01:08.678427 4077 raft_consensus.cc:2454] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df655d53530449fc85de33de1c229815 in term 2.
W20240502 14:01:08.682360 4060 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59796: Service unavailable: leader transfer in progress
W20240502 14:01:08.690124 4060 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59796: Illegal state: replica 24fd5d6c84c1483ea8f39bb975e26e45 is not leader of this config: current role FOLLOWER
I20240502 14:01:08.693508 4219 raft_consensus.cc:979] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45: : Instructing follower df655d53530449fc85de33de1c229815 to start an election
I20240502 14:01:08.693586 4219 raft_consensus.cc:1067] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Signalling peer df655d53530449fc85de33de1c229815 to start an election
I20240502 14:01:08.702615 4150 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c965f0e11bc1433f8ccdc20c98b4b4b9"
dest_uuid: "df655d53530449fc85de33de1c229815"
from {username='slave'} at 127.0.0.1:33578
I20240502 14:01:08.702771 4150 raft_consensus.cc:482] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:08.702881 4150 raft_consensus.cc:3037] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:08.703827 4150 raft_consensus.cc:504] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:08.704393 4001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c965f0e11bc1433f8ccdc20c98b4b4b9" candidate_uuid: "df655d53530449fc85de33de1c229815" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: true dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
I20240502 14:01:08.704488 4001 raft_consensus.cc:3037] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:08.705690 4001 raft_consensus.cc:2454] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df655d53530449fc85de33de1c229815 in term 2.
I20240502 14:01:08.705904 4122 leader_election.cc:302] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd060dd4339f4fa5a985596bf7dec0ae, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:08.706059 4351 raft_consensus.cc:2781] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:08.706159 4351 raft_consensus.cc:686] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 2 LEADER]: Becoming Leader. State: Replica: df655d53530449fc85de33de1c229815, State: Running, Role: LEADER
I20240502 14:01:08.706295 4150 leader_election.cc:288] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [CANDIDATE]: Term 2 election: Requested vote from peers bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667), 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251)
I20240502 14:01:08.706281 4351 consensus_queue.cc:237] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.23, Last appended by leader: 23, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
W20240502 14:01:08.707060 4060 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59796: Service unavailable: leader transfer in progress
I20240502 14:01:08.707228 4078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c965f0e11bc1433f8ccdc20c98b4b4b9" candidate_uuid: "df655d53530449fc85de33de1c229815" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: true dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
I20240502 14:01:08.707334 4078 raft_consensus.cc:3032] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:01:08.707372 4078 raft_consensus.cc:729] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Running, Role: LEADER
I20240502 14:01:08.707486 4078 consensus_queue.cc:260] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.23, Last appended by leader: 23, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:08.707623 4078 raft_consensus.cc:3037] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:08.708465 4078 raft_consensus.cc:2454] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df655d53530449fc85de33de1c229815 in term 2.
W20240502 14:01:08.710228 4060 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59796: Illegal state: replica 24fd5d6c84c1483ea8f39bb975e26e45 is not leader of this config: current role FOLLOWER
W20240502 14:01:08.710312 4059 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59796: Illegal state: replica 24fd5d6c84c1483ea8f39bb975e26e45 is not leader of this config: current role FOLLOWER
W20240502 14:01:08.712940 4057 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59796: Illegal state: replica 24fd5d6c84c1483ea8f39bb975e26e45 is not leader of this config: current role FOLLOWER
I20240502 14:01:08.714762 4078 raft_consensus.cc:1261] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 2 index: 24. (index mismatch)
I20240502 14:01:08.715067 4001 raft_consensus.cc:1261] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 2 index: 24. (index mismatch)
I20240502 14:01:08.715695 4349 consensus_queue.cc:1035] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20240502 14:01:08.716015 4347 consensus_queue.cc:1035] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
W20240502 14:01:08.717211 4057 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59796: Illegal state: replica 24fd5d6c84c1483ea8f39bb975e26e45 is not leader of this config: current role FOLLOWER
W20240502 14:01:08.721063 4060 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59796: Illegal state: replica 24fd5d6c84c1483ea8f39bb975e26e45 is not leader of this config: current role FOLLOWER
W20240502 14:01:08.722745 4060 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59796: Illegal state: replica 24fd5d6c84c1483ea8f39bb975e26e45 is not leader of this config: current role FOLLOWER
W20240502 14:01:08.732057 3977 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57076: Illegal state: replica bd060dd4339f4fa5a985596bf7dec0ae is not leader of this config: current role FOLLOWER
W20240502 14:01:08.732523 3981 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57076: Illegal state: replica bd060dd4339f4fa5a985596bf7dec0ae is not leader of this config: current role FOLLOWER
W20240502 14:01:08.733162 3981 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57076: Illegal state: replica bd060dd4339f4fa5a985596bf7dec0ae is not leader of this config: current role FOLLOWER
W20240502 14:01:08.733304 4060 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59796: Illegal state: replica 24fd5d6c84c1483ea8f39bb975e26e45 is not leader of this config: current role FOLLOWER
I20240502 14:01:08.742784 4078 raft_consensus.cc:1261] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20240502 14:01:08.743081 4348 consensus_queue.cc:1035] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20240502 14:01:08.743398 4001 raft_consensus.cc:1261] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20240502 14:01:08.743700 4351 consensus_queue.cc:1035] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20240502 14:01:08.752588 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:01:08.755165 4356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:08.755508 4354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:08.756450 4353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:01:08.757237 581 server_base.cc:1034] running on GCE node
I20240502 14:01:08.757519 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:01:08.757566 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:01:08.757586 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658468757586 us; error 0 us; skew 500 ppm
I20240502 14:01:08.758216 3900 catalog_manager.cc:5484] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 reported cstate change: term changed from 1 to 2, leader changed from 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66) to df655d53530449fc85de33de1c229815 (127.0.145.67). New cstate: current_term: 2 leader_uuid: "df655d53530449fc85de33de1c229815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: HEALTHY } } }
I20240502 14:01:08.758612 581 webserver.cc:460] Webserver started at http://127.0.145.68:34745/ using document root <none> and password file <none>
I20240502 14:01:08.758741 581 fs_manager.cc:359] Metadata directory not provided
I20240502 14:01:08.758783 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:01:08.758844 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:01:08.759222 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-3-root/instance:
uuid: "5af426c57bc3434c9fbdcdff873a565c"
format_stamp: "Formatted at 2024-05-02 14:01:08 on dist-test-slave-1z5j"
I20240502 14:01:08.760843 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 14:01:08.761626 4362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:08.761822 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:08.761896 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-3-root
uuid: "5af426c57bc3434c9fbdcdff873a565c"
format_stamp: "Formatted at 2024-05-02 14:01:08 on dist-test-slave-1z5j"
I20240502 14:01:08.761960 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:01:08.768100 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:01:08.768353 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:01:08.768575 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:01:08.769035 581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:01:08.769073 581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:08.769109 581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:01:08.769137 581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:08.773618 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.68:34601
I20240502 14:01:08.773650 4426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.68:34601 every 8 connection(s)
I20240502 14:01:08.774309 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:01:08.775983 4431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:08.776387 4432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:01:08.776880 581 server_base.cc:1034] running on GCE node
W20240502 14:01:08.777268 4434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:01:08.777462 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:01:08.777525 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:01:08.777547 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658468777547 us; error 0 us; skew 500 ppm
I20240502 14:01:08.777946 4427 heartbeater.cc:347] Connected to a master server at 127.0.145.126:37069
I20240502 14:01:08.778030 4427 heartbeater.cc:464] Registering TS with master...
I20240502 14:01:08.778235 4427 heartbeater.cc:511] Master 127.0.145.126:37069 requested a full tablet report, sending...
I20240502 14:01:08.778483 581 webserver.cc:460] Webserver started at http://127.0.145.69:34055/ using document root <none> and password file <none>
I20240502 14:01:08.778610 581 fs_manager.cc:359] Metadata directory not provided
I20240502 14:01:08.778662 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:01:08.778709 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:01:08.778777 3903 ts_manager.cc:194] Registered new tserver with Master: 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68:34601)
I20240502 14:01:08.779018 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-4-root/instance:
uuid: "b3988f2f38654eedac11fa475e56e92f"
format_stamp: "Formatted at 2024-05-02 14:01:08 on dist-test-slave-1z5j"
I20240502 14:01:08.779238 3903 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40862
I20240502 14:01:08.780350 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 14:01:08.781131 4439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:08.781359 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:08.781421 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-4-root
uuid: "b3988f2f38654eedac11fa475e56e92f"
format_stamp: "Formatted at 2024-05-02 14:01:08 on dist-test-slave-1z5j"
I20240502 14:01:08.781483 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-4-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-4-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-4-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:01:08.788518 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:01:08.788829 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:01:08.789084 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:01:08.789428 581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:01:08.789463 581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:08.789510 581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:01:08.789534 581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:08.793713 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.69:45021
I20240502 14:01:08.793743 4503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.69:45021 every 8 connection(s)
I20240502 14:01:08.794339 581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:01:08.795861 4507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:08.796298 4508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:08.796675 4510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:01:08.797122 581 server_base.cc:1034] running on GCE node
I20240502 14:01:08.797266 581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:01:08.797317 581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:01:08.797349 581 hybrid_clock.cc:648] HybridClock initialized: now 1714658468797348 us; error 0 us; skew 500 ppm
I20240502 14:01:08.798241 581 webserver.cc:460] Webserver started at http://127.0.145.70:33749/ using document root <none> and password file <none>
I20240502 14:01:08.798369 581 fs_manager.cc:359] Metadata directory not provided
I20240502 14:01:08.798422 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:01:08.798436 4504 heartbeater.cc:347] Connected to a master server at 127.0.145.126:37069
I20240502 14:01:08.798496 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:01:08.798523 4504 heartbeater.cc:464] Registering TS with master...
I20240502 14:01:08.798692 4504 heartbeater.cc:511] Master 127.0.145.126:37069 requested a full tablet report, sending...
I20240502 14:01:08.798796 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-5-root/instance:
uuid: "e1de31532912433dbed75c2bb01bc42a"
format_stamp: "Formatted at 2024-05-02 14:01:08 on dist-test-slave-1z5j"
I20240502 14:01:08.799193 3903 ts_manager.cc:194] Registered new tserver with Master: b3988f2f38654eedac11fa475e56e92f (127.0.145.69:45021)
I20240502 14:01:08.799614 3903 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40872
I20240502 14:01:08.800144 581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240502 14:01:08.801016 4515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:08.801237 581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:08.801297 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-5-root
uuid: "e1de31532912433dbed75c2bb01bc42a"
format_stamp: "Formatted at 2024-05-02 14:01:08 on dist-test-slave-1z5j"
I20240502 14:01:08.801357 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-5-root
metadata directory: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-5-root
1 data directories: /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-5-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:01:08.818917 581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:01:08.819216 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:01:08.819449 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:01:08.819829 581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:01:08.819869 581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:08.819904 581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:01:08.819933 581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:08.824574 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.70:41637
I20240502 14:01:08.824653 4579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.70:41637 every 8 connection(s)
I20240502 14:01:08.828191 4580 heartbeater.cc:347] Connected to a master server at 127.0.145.126:37069
I20240502 14:01:08.828267 4580 heartbeater.cc:464] Registering TS with master...
I20240502 14:01:08.828466 4580 heartbeater.cc:511] Master 127.0.145.126:37069 requested a full tablet report, sending...
I20240502 14:01:08.828943 3903 ts_manager.cc:194] Registered new tserver with Master: e1de31532912433dbed75c2bb01bc42a (127.0.145.70:41637)
I20240502 14:01:08.829456 3903 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40874
I20240502 14:01:09.638948 4150 consensus_queue.cc:237] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 22, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:09.640163 4005 raft_consensus.cc:1261] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 14:01:09.640240 4078 raft_consensus.cc:1261] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 14:01:09.640556 4351 consensus_queue.cc:1035] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:01:09.640743 4586 consensus_queue.cc:1035] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:01:09.642306 4586 raft_consensus.cc:2932] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER b3988f2f38654eedac11fa475e56e92f (127.0.145.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } } }
I20240502 14:01:09.642568 4078 raft_consensus.cc:2932] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER b3988f2f38654eedac11fa475e56e92f (127.0.145.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } } }
I20240502 14:01:09.642642 4005 raft_consensus.cc:2932] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER b3988f2f38654eedac11fa475e56e92f (127.0.145.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } } }
W20240502 14:01:09.644163 4120 consensus_peers.cc:485] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 -> Peer b3988f2f38654eedac11fa475e56e92f (127.0.145.69:45021): Couldn't send request to peer b3988f2f38654eedac11fa475e56e92f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7908edf8bbe8419ab21d0c93cd2a3b35. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:09.645229 3903 catalog_manager.cc:5484] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae reported cstate change: config changed from index -1 to 28, NON_VOTER b3988f2f38654eedac11fa475e56e92f (127.0.145.69) added. New cstate: current_term: 2 leader_uuid: "df655d53530449fc85de33de1c229815" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } } }
I20240502 14:01:09.648543 4078 consensus_queue.cc:237] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } }
I20240502 14:01:09.649663 4005 raft_consensus.cc:1261] T a4d0e9d6d8e7445a9367e0f3a31c8748 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 14:01:09.649869 4150 raft_consensus.cc:1261] T a4d0e9d6d8e7445a9367e0f3a31c8748 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 14:01:09.650148 4594 consensus_queue.cc:1035] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:01:09.650331 4593 consensus_queue.cc:1035] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:01:09.651660 4287 raft_consensus.cc:2932] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER e1de31532912433dbed75c2bb01bc42a (127.0.145.70) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } } }
I20240502 14:01:09.652030 4150 raft_consensus.cc:2932] T a4d0e9d6d8e7445a9367e0f3a31c8748 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER e1de31532912433dbed75c2bb01bc42a (127.0.145.70) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } } }
I20240502 14:01:09.652148 4005 raft_consensus.cc:2932] T a4d0e9d6d8e7445a9367e0f3a31c8748 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER e1de31532912433dbed75c2bb01bc42a (127.0.145.70) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } } }
I20240502 14:01:09.654049 3898 catalog_manager.cc:5484] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 reported cstate change: config changed from index -1 to 27, NON_VOTER e1de31532912433dbed75c2bb01bc42a (127.0.145.70) added. New cstate: current_term: 1 leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:01:09.655345 4045 consensus_peers.cc:485] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 -> Peer e1de31532912433dbed75c2bb01bc42a (127.0.145.70:41637): Couldn't send request to peer e1de31532912433dbed75c2bb01bc42a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4d0e9d6d8e7445a9367e0f3a31c8748. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:09.658625 4005 consensus_queue.cc:237] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } }
I20240502 14:01:09.659705 4078 raft_consensus.cc:1261] T e6ca4998e3634ba789210e0c072e29b0 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 14:01:09.660022 4150 raft_consensus.cc:1261] T e6ca4998e3634ba789210e0c072e29b0 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 14:01:09.660063 4598 consensus_queue.cc:1035] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:01:09.660332 4599 consensus_queue.cc:1035] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:01:09.661697 4292 raft_consensus.cc:2932] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } } }
I20240502 14:01:09.663429 4150 raft_consensus.cc:2932] T e6ca4998e3634ba789210e0c072e29b0 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } } }
I20240502 14:01:09.663772 3903 catalog_manager.cc:5484] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae reported cstate change: config changed from index -1 to 28, NON_VOTER 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) added. New cstate: current_term: 1 leader_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:09.663831 4078 consensus_queue.cc:237] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } }
W20240502 14:01:09.665014 3972 consensus_peers.cc:485] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae -> Peer 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68:34601): Couldn't send request to peer 5af426c57bc3434c9fbdcdff873a565c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6ca4998e3634ba789210e0c072e29b0. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:09.665344 4150 raft_consensus.cc:1261] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 14:01:09.665657 4594 consensus_queue.cc:1035] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:01:09.665760 4005 raft_consensus.cc:1261] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 14:01:09.665747 4078 raft_consensus.cc:2932] T e6ca4998e3634ba789210e0c072e29b0 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } } }
I20240502 14:01:09.666123 4593 consensus_queue.cc:1035] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:01:09.667224 4287 raft_consensus.cc:2932] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } } }
I20240502 14:01:09.667457 4150 raft_consensus.cc:2932] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } } }
I20240502 14:01:09.669404 4078 consensus_queue.cc:237] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } }
I20240502 14:01:09.669445 3903 catalog_manager.cc:5484] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 reported cstate change: config changed from index -1 to 27, NON_VOTER 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) added. New cstate: current_term: 1 leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:09.670045 4005 raft_consensus.cc:2932] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } } }
I20240502 14:01:09.670823 4150 raft_consensus.cc:1261] T dba8942501c440b18f71c7106878d0c4 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
W20240502 14:01:09.670987 4045 consensus_peers.cc:485] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 -> Peer e1de31532912433dbed75c2bb01bc42a (127.0.145.70:41637): Couldn't send request to peer e1de31532912433dbed75c2bb01bc42a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dba8942501c440b18f71c7106878d0c4. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:09.671092 4001 raft_consensus.cc:1261] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 14:01:09.671113 4594 consensus_queue.cc:1035] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:01:09.671411 4594 consensus_queue.cc:1035] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:01:09.672477 4593 raft_consensus.cc:2932] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER e1de31532912433dbed75c2bb01bc42a (127.0.145.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } } }
I20240502 14:01:09.674378 4001 raft_consensus.cc:2932] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER e1de31532912433dbed75c2bb01bc42a (127.0.145.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } } }
I20240502 14:01:09.674815 4150 raft_consensus.cc:2932] T dba8942501c440b18f71c7106878d0c4 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER e1de31532912433dbed75c2bb01bc42a (127.0.145.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } } }
I20240502 14:01:09.675233 3900 catalog_manager.cc:5484] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 reported cstate change: config changed from index -1 to 28, NON_VOTER e1de31532912433dbed75c2bb01bc42a (127.0.145.70) added. New cstate: current_term: 1 leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:09.675969 4005 consensus_queue.cc:237] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:09.678438 4150 raft_consensus.cc:1261] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 14:01:09.678992 4078 raft_consensus.cc:1261] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 14:01:09.679078 4598 consensus_queue.cc:1035] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 14:01:09.679320 4602 consensus_queue.cc:1035] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 14:01:09.680783 4292 raft_consensus.cc:2932] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER b3988f2f38654eedac11fa475e56e92f (127.0.145.69) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } } }
W20240502 14:01:09.681450 4048 consensus_peers.cc:485] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 -> Peer 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68:34601): Couldn't send request to peer 5af426c57bc3434c9fbdcdff873a565c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d0d4aeaec264d04bae2c34e6edc6ec7. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:09.681519 4078 raft_consensus.cc:2932] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER b3988f2f38654eedac11fa475e56e92f (127.0.145.69) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } } }
W20240502 14:01:09.682299 3969 consensus_peers.cc:485] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae -> Peer b3988f2f38654eedac11fa475e56e92f (127.0.145.69:45021): Couldn't send request to peer b3988f2f38654eedac11fa475e56e92f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5172e93df3a4fb5a70188b0507639e6. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:09.683122 3898 catalog_manager.cc:5484] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae reported cstate change: config changed from index -1 to 26, NON_VOTER b3988f2f38654eedac11fa475e56e92f (127.0.145.69) added. New cstate: current_term: 1 leader_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:09.683153 4150 raft_consensus.cc:2932] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER b3988f2f38654eedac11fa475e56e92f (127.0.145.69) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } } }
I20240502 14:01:09.780015 4427 heartbeater.cc:503] Master 127.0.145.126:37069 was elected leader, sending a full tablet report...
I20240502 14:01:09.800299 4504 heartbeater.cc:503] Master 127.0.145.126:37069 was elected leader, sending a full tablet report...
I20240502 14:01:09.830152 4580 heartbeater.cc:503] Master 127.0.145.126:37069 was elected leader, sending a full tablet report...
I20240502 14:01:10.037663 4605 ts_tablet_manager.cc:902] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f: Initiating tablet copy from peer df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:10.038201 4605 tablet_copy_client.cc:308] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.67:34309
I20240502 14:01:10.041858 4167 tablet_copy_service.cc:140] P df655d53530449fc85de33de1c229815: Received BeginTabletCopySession request for tablet 7908edf8bbe8419ab21d0c93cd2a3b35 from peer b3988f2f38654eedac11fa475e56e92f ({username='slave'} at 127.0.0.1:33622)
I20240502 14:01:10.041965 4167 tablet_copy_service.cc:161] P df655d53530449fc85de33de1c229815: Beginning new tablet copy session on tablet 7908edf8bbe8419ab21d0c93cd2a3b35 from peer b3988f2f38654eedac11fa475e56e92f at {username='slave'} at 127.0.0.1:33622: session id = b3988f2f38654eedac11fa475e56e92f-7908edf8bbe8419ab21d0c93cd2a3b35
I20240502 14:01:10.042840 4167 tablet_copy_source_session.cc:205] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:10.043574 4605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7908edf8bbe8419ab21d0c93cd2a3b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:10.045728 4605 tablet_copy_client.cc:783] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:10.045872 4605 tablet_copy_client.cc:647] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:10.046908 4609 tablet_copy_client.cc:1002] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f: tablet copy: Injecting latency into file download: 2000
I20240502 14:01:10.069666 4611 ts_tablet_manager.cc:902] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f: Initiating tablet copy from peer bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667)
I20240502 14:01:10.070235 4611 tablet_copy_client.cc:308] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.65:46667
I20240502 14:01:10.073475 4016 tablet_copy_service.cc:140] P bd060dd4339f4fa5a985596bf7dec0ae: Received BeginTabletCopySession request for tablet b5172e93df3a4fb5a70188b0507639e6 from peer b3988f2f38654eedac11fa475e56e92f ({username='slave'} at 127.0.0.1:57094)
I20240502 14:01:10.073557 4016 tablet_copy_service.cc:161] P bd060dd4339f4fa5a985596bf7dec0ae: Beginning new tablet copy session on tablet b5172e93df3a4fb5a70188b0507639e6 from peer b3988f2f38654eedac11fa475e56e92f at {username='slave'} at 127.0.0.1:57094: session id = b3988f2f38654eedac11fa475e56e92f-b5172e93df3a4fb5a70188b0507639e6
I20240502 14:01:10.074358 4016 tablet_copy_source_session.cc:205] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:10.075069 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5172e93df3a4fb5a70188b0507639e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:10.076951 4611 tablet_copy_client.cc:783] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:10.077086 4611 tablet_copy_client.cc:647] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:10.077972 4615 tablet_copy_client.cc:1002] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f: tablet copy: Injecting latency into file download: 2000
I20240502 14:01:10.118942 4616 ts_tablet_manager.cc:902] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 5af426c57bc3434c9fbdcdff873a565c: Initiating tablet copy from peer 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251)
I20240502 14:01:10.119539 4616 tablet_copy_client.cc:308] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 5af426c57bc3434c9fbdcdff873a565c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.66:35251
I20240502 14:01:10.123057 4092 tablet_copy_service.cc:140] P 24fd5d6c84c1483ea8f39bb975e26e45: Received BeginTabletCopySession request for tablet 3d0d4aeaec264d04bae2c34e6edc6ec7 from peer 5af426c57bc3434c9fbdcdff873a565c ({username='slave'} at 127.0.0.1:59816)
I20240502 14:01:10.123139 4092 tablet_copy_service.cc:161] P 24fd5d6c84c1483ea8f39bb975e26e45: Beginning new tablet copy session on tablet 3d0d4aeaec264d04bae2c34e6edc6ec7 from peer 5af426c57bc3434c9fbdcdff873a565c at {username='slave'} at 127.0.0.1:59816: session id = 5af426c57bc3434c9fbdcdff873a565c-3d0d4aeaec264d04bae2c34e6edc6ec7
I20240502 14:01:10.123865 4092 tablet_copy_source_session.cc:205] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:10.124603 4616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d0d4aeaec264d04bae2c34e6edc6ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:10.126438 4616 tablet_copy_client.cc:783] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 5af426c57bc3434c9fbdcdff873a565c: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:10.126578 4616 tablet_copy_client.cc:647] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 5af426c57bc3434c9fbdcdff873a565c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:10.127748 4620 tablet_copy_client.cc:1002] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 5af426c57bc3434c9fbdcdff873a565c: tablet copy: Injecting latency into file download: 2000
I20240502 14:01:10.142109 4622 ts_tablet_manager.cc:902] T a4d0e9d6d8e7445a9367e0f3a31c8748 P e1de31532912433dbed75c2bb01bc42a: Initiating tablet copy from peer 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251)
I20240502 14:01:10.142622 4622 tablet_copy_client.cc:308] T a4d0e9d6d8e7445a9367e0f3a31c8748 P e1de31532912433dbed75c2bb01bc42a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.66:35251
I20240502 14:01:10.146035 4092 tablet_copy_service.cc:140] P 24fd5d6c84c1483ea8f39bb975e26e45: Received BeginTabletCopySession request for tablet a4d0e9d6d8e7445a9367e0f3a31c8748 from peer e1de31532912433dbed75c2bb01bc42a ({username='slave'} at 127.0.0.1:59820)
I20240502 14:01:10.146116 4092 tablet_copy_service.cc:161] P 24fd5d6c84c1483ea8f39bb975e26e45: Beginning new tablet copy session on tablet a4d0e9d6d8e7445a9367e0f3a31c8748 from peer e1de31532912433dbed75c2bb01bc42a at {username='slave'} at 127.0.0.1:59820: session id = e1de31532912433dbed75c2bb01bc42a-a4d0e9d6d8e7445a9367e0f3a31c8748
I20240502 14:01:10.146827 4092 tablet_copy_source_session.cc:205] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:10.147480 4622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4d0e9d6d8e7445a9367e0f3a31c8748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:10.149142 4622 tablet_copy_client.cc:783] T a4d0e9d6d8e7445a9367e0f3a31c8748 P e1de31532912433dbed75c2bb01bc42a: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:10.149279 4622 tablet_copy_client.cc:647] T a4d0e9d6d8e7445a9367e0f3a31c8748 P e1de31532912433dbed75c2bb01bc42a: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:10.150074 4626 tablet_copy_client.cc:1002] T a4d0e9d6d8e7445a9367e0f3a31c8748 P e1de31532912433dbed75c2bb01bc42a: tablet copy: Injecting latency into file download: 2000
I20240502 14:01:10.266732 4628 ts_tablet_manager.cc:902] T dba8942501c440b18f71c7106878d0c4 P e1de31532912433dbed75c2bb01bc42a: Initiating tablet copy from peer 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251)
I20240502 14:01:10.267374 4628 tablet_copy_client.cc:308] T dba8942501c440b18f71c7106878d0c4 P e1de31532912433dbed75c2bb01bc42a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.66:35251
I20240502 14:01:10.267796 4092 tablet_copy_service.cc:140] P 24fd5d6c84c1483ea8f39bb975e26e45: Received BeginTabletCopySession request for tablet dba8942501c440b18f71c7106878d0c4 from peer e1de31532912433dbed75c2bb01bc42a ({username='slave'} at 127.0.0.1:59820)
I20240502 14:01:10.267894 4092 tablet_copy_service.cc:161] P 24fd5d6c84c1483ea8f39bb975e26e45: Beginning new tablet copy session on tablet dba8942501c440b18f71c7106878d0c4 from peer e1de31532912433dbed75c2bb01bc42a at {username='slave'} at 127.0.0.1:59820: session id = e1de31532912433dbed75c2bb01bc42a-dba8942501c440b18f71c7106878d0c4
I20240502 14:01:10.268658 4092 tablet_copy_source_session.cc:205] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:10.269454 4628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dba8942501c440b18f71c7106878d0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:10.271545 4628 tablet_copy_client.cc:783] T dba8942501c440b18f71c7106878d0c4 P e1de31532912433dbed75c2bb01bc42a: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:10.271708 4628 tablet_copy_client.cc:647] T dba8942501c440b18f71c7106878d0c4 P e1de31532912433dbed75c2bb01bc42a: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:10.272656 4632 tablet_copy_client.cc:1002] T dba8942501c440b18f71c7106878d0c4 P e1de31532912433dbed75c2bb01bc42a: tablet copy: Injecting latency into file download: 2000
I20240502 14:01:10.272994 4633 ts_tablet_manager.cc:902] T e6ca4998e3634ba789210e0c072e29b0 P 5af426c57bc3434c9fbdcdff873a565c: Initiating tablet copy from peer bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667)
I20240502 14:01:10.273492 4633 tablet_copy_client.cc:308] T e6ca4998e3634ba789210e0c072e29b0 P 5af426c57bc3434c9fbdcdff873a565c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.65:46667
I20240502 14:01:10.276904 4016 tablet_copy_service.cc:140] P bd060dd4339f4fa5a985596bf7dec0ae: Received BeginTabletCopySession request for tablet e6ca4998e3634ba789210e0c072e29b0 from peer 5af426c57bc3434c9fbdcdff873a565c ({username='slave'} at 127.0.0.1:57102)
I20240502 14:01:10.276993 4016 tablet_copy_service.cc:161] P bd060dd4339f4fa5a985596bf7dec0ae: Beginning new tablet copy session on tablet e6ca4998e3634ba789210e0c072e29b0 from peer 5af426c57bc3434c9fbdcdff873a565c at {username='slave'} at 127.0.0.1:57102: session id = 5af426c57bc3434c9fbdcdff873a565c-e6ca4998e3634ba789210e0c072e29b0
I20240502 14:01:10.277787 4016 tablet_copy_source_session.cc:205] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:10.278421 4633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6ca4998e3634ba789210e0c072e29b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:10.280089 4633 tablet_copy_client.cc:783] T e6ca4998e3634ba789210e0c072e29b0 P 5af426c57bc3434c9fbdcdff873a565c: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:10.280305 4633 tablet_copy_client.cc:647] T e6ca4998e3634ba789210e0c072e29b0 P 5af426c57bc3434c9fbdcdff873a565c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:10.281097 4637 tablet_copy_client.cc:1002] T e6ca4998e3634ba789210e0c072e29b0 P 5af426c57bc3434c9fbdcdff873a565c: tablet copy: Injecting latency into file download: 2000
I20240502 14:01:10.662124 3957 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:01:10.667784 4150 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "144739b71bf6474d90399816120b3b59"
dest_uuid: "df655d53530449fc85de33de1c229815"
mode: GRACEFUL
new_leader_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
from {username='slave'} at 127.0.0.1:33630
I20240502 14:01:10.667924 4150 raft_consensus.cc:595] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 1 LEADER]: Received request to transfer leadership to TS bd060dd4339f4fa5a985596bf7dec0ae
I20240502 14:01:10.668828 4001 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b5172e93df3a4fb5a70188b0507639e6"
dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
mode: GRACEFUL
new_leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
from {username='slave'} at 127.0.0.1:57080
I20240502 14:01:10.668939 4001 raft_consensus.cc:595] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 LEADER]: Received request to transfer leadership to TS 24fd5d6c84c1483ea8f39bb975e26e45
I20240502 14:01:10.669243 3957 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240502 14:01:10.669350 3957 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:01:10.770332 4638 raft_consensus.cc:979] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815: : Instructing follower bd060dd4339f4fa5a985596bf7dec0ae to start an election
I20240502 14:01:10.770464 4638 raft_consensus.cc:1067] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 1 LEADER]: Signalling peer bd060dd4339f4fa5a985596bf7dec0ae to start an election
I20240502 14:01:10.770848 4001 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "144739b71bf6474d90399816120b3b59"
dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
from {username='slave'} at 127.0.0.1:57070
I20240502 14:01:10.770962 4001 raft_consensus.cc:482] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:10.771005 4001 raft_consensus.cc:3037] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:10.771775 4001 raft_consensus.cc:504] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:10.772071 4001 leader_election.cc:288] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 2 election: Requested vote from peers 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:10.772336 4082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144739b71bf6474d90399816120b3b59" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
I20240502 14:01:10.772364 4150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144739b71bf6474d90399816120b3b59" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "df655d53530449fc85de33de1c229815"
I20240502 14:01:10.772444 4082 raft_consensus.cc:3037] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:10.772483 4150 raft_consensus.cc:3032] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:01:10.772547 4150 raft_consensus.cc:729] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df655d53530449fc85de33de1c229815, State: Running, Role: LEADER
I20240502 14:01:10.772671 4150 consensus_queue.cc:260] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:10.772821 4150 raft_consensus.cc:3037] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:10.773309 4082 raft_consensus.cc:2454] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 2.
I20240502 14:01:10.773576 4150 raft_consensus.cc:2454] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 2.
I20240502 14:01:10.773582 3972 leader_election.cc:302] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, bd060dd4339f4fa5a985596bf7dec0ae; no voters:
I20240502 14:01:10.773759 4642 raft_consensus.cc:2781] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:10.773840 4642 raft_consensus.cc:686] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 LEADER]: Becoming Leader. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Running, Role: LEADER
I20240502 14:01:10.773959 4642 consensus_queue.cc:237] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:10.775446 3903 catalog_manager.cc:5484] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae reported cstate change: term changed from 1 to 2, leader changed from df655d53530449fc85de33de1c229815 (127.0.145.67) to bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65). New cstate: current_term: 2 leader_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:10.804581 4642 raft_consensus.cc:979] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae: : Instructing follower 24fd5d6c84c1483ea8f39bb975e26e45 to start an election
I20240502 14:01:10.804684 4642 raft_consensus.cc:1067] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 LEADER]: Signalling peer 24fd5d6c84c1483ea8f39bb975e26e45 to start an election
I20240502 14:01:10.805078 4082 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b5172e93df3a4fb5a70188b0507639e6"
dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
from {username='slave'} at 127.0.0.1:59794
I20240502 14:01:10.805212 4082 raft_consensus.cc:482] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:10.805279 4082 raft_consensus.cc:3037] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:10.806159 4082 raft_consensus.cc:504] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:10.806522 4082 leader_election.cc:288] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 2 election: Requested vote from peers bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:10.806738 4001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5172e93df3a4fb5a70188b0507639e6" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
I20240502 14:01:10.806874 4150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5172e93df3a4fb5a70188b0507639e6" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "df655d53530449fc85de33de1c229815"
I20240502 14:01:10.807020 4001 raft_consensus.cc:3032] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:01:10.807037 4150 raft_consensus.cc:3037] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:10.807116 4001 raft_consensus.cc:729] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Running, Role: LEADER
I20240502 14:01:10.807253 4001 consensus_queue.cc:260] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:10.807417 4001 raft_consensus.cc:3037] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:10.807945 4150 raft_consensus.cc:2454] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 2.
I20240502 14:01:10.808153 4048 leader_election.cc:302] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:10.808187 4001 raft_consensus.cc:2454] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 2.
I20240502 14:01:10.808319 4594 raft_consensus.cc:2781] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:10.808410 4594 raft_consensus.cc:686] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 LEADER]: Becoming Leader. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Running, Role: LEADER
I20240502 14:01:10.808535 4594 consensus_queue.cc:237] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:10.810032 3903 catalog_manager.cc:5484] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 reported cstate change: term changed from 1 to 2, leader changed from bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65) to 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66). New cstate: current_term: 2 leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:10.815913 4082 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b5172e93df3a4fb5a70188b0507639e6"
dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59772
I20240502 14:01:10.816025 4082 raft_consensus.cc:595] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 LEADER]: Received request to transfer leadership
I20240502 14:01:10.820129 4082 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b5172e93df3a4fb5a70188b0507639e6"
dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59772
I20240502 14:01:10.820219 4082 raft_consensus.cc:595] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 LEADER]: Received request to transfer leadership
W20240502 14:01:10.820456 3890 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for b5172e93df3a4fb5a70188b0507639e6 already in progress
I20240502 14:01:10.821090 4082 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b5172e93df3a4fb5a70188b0507639e6"
dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59772
I20240502 14:01:10.821177 4082 raft_consensus.cc:595] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 LEADER]: Received request to transfer leadership
W20240502 14:01:10.821466 3952 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for b5172e93df3a4fb5a70188b0507639e6 already in progress
W20240502 14:01:10.821518 3952 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for b5172e93df3a4fb5a70188b0507639e6 already in progress
I20240502 14:01:11.153010 4150 raft_consensus.cc:1261] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 2 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 14:01:11.153348 4642 consensus_queue.cc:1035] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:01:11.156116 4078 raft_consensus.cc:1261] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 14:01:11.156474 4641 consensus_queue.cc:1035] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:01:11.197305 4005 raft_consensus.cc:1261] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 14:01:11.197657 4640 consensus_queue.cc:1035] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:01:11.199070 4594 raft_consensus.cc:979] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45: : Instructing follower bd060dd4339f4fa5a985596bf7dec0ae to start an election
I20240502 14:01:11.199175 4643 raft_consensus.cc:1067] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 LEADER]: Signalling peer bd060dd4339f4fa5a985596bf7dec0ae to start an election
I20240502 14:01:11.200336 4000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b5172e93df3a4fb5a70188b0507639e6"
dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
from {username='slave'} at 127.0.0.1:57072
I20240502 14:01:11.200461 4000 raft_consensus.cc:482] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:11.200520 4000 raft_consensus.cc:3037] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:01:11.201718 4000 raft_consensus.cc:504] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:11.202083 4000 leader_election.cc:288] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 3 election: Requested vote from peers 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:11.202391 4150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5172e93df3a4fb5a70188b0507639e6" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "df655d53530449fc85de33de1c229815"
I20240502 14:01:11.202525 4150 raft_consensus.cc:3037] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:01:11.203433 4150 raft_consensus.cc:2454] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 3.
I20240502 14:01:11.202440 4082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5172e93df3a4fb5a70188b0507639e6" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
I20240502 14:01:11.203641 4082 raft_consensus.cc:3032] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:01:11.203676 4082 raft_consensus.cc:729] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Running, Role: LEADER
I20240502 14:01:11.203707 3972 leader_election.cc:302] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd060dd4339f4fa5a985596bf7dec0ae, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:11.203797 4082 consensus_queue.cc:260] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:11.203857 4641 raft_consensus.cc:2781] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:01:11.203927 4641 raft_consensus.cc:686] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 LEADER]: Becoming Leader. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Running, Role: LEADER
I20240502 14:01:11.203965 4082 raft_consensus.cc:3037] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:01:11.204048 4641 consensus_queue.cc:237] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:11.204854 4082 raft_consensus.cc:2454] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 3.
I20240502 14:01:11.204988 4150 raft_consensus.cc:1226] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815 [term 3 FOLLOWER]: Rejecting Update request from peer 24fd5d6c84c1483ea8f39bb975e26e45 for earlier term 2. Current term is 3. Ops: []
I20240502 14:01:11.205562 3903 catalog_manager.cc:5484] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae reported cstate change: term changed from 2 to 3, leader changed from 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66) to bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65). New cstate: current_term: 3 leader_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:01:11.671762 3969 consensus_peers.cc:485] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae -> Peer b3988f2f38654eedac11fa475e56e92f (127.0.145.69:45021): Couldn't send request to peer b3988f2f38654eedac11fa475e56e92f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:11.672885 4082 raft_consensus.cc:1261] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 14:01:11.673193 4641 consensus_queue.cc:1035] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:01:11.675451 4150 raft_consensus.cc:1261] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815 [term 3 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 14:01:11.676306 4602 consensus_queue.cc:1035] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:01:12.047187 4605 tablet_copy_client.cc:523] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:12.048580 4605 tablet_bootstrap.cc:492] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f: Bootstrap starting.
I20240502 14:01:12.069042 4605 tablet_bootstrap.cc:492] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=109 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:12.069298 4605 tablet_bootstrap.cc:492] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f: Bootstrap complete.
I20240502 14:01:12.069464 4605 ts_tablet_manager.cc:1367] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f: Time spent bootstrapping tablet: real 0.021s user 0.018s sys 0.004s
I20240502 14:01:12.069888 4605 raft_consensus.cc:348] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:12.070137 4605 raft_consensus.cc:729] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b3988f2f38654eedac11fa475e56e92f, State: Initialized, Role: LEARNER
I20240502 14:01:12.070257 4605 consensus_queue.cc:260] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:12.070598 4605 ts_tablet_manager.cc:1398] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:12.071002 4167 tablet_copy_service.cc:342] P df655d53530449fc85de33de1c229815: Request end of tablet copy session b3988f2f38654eedac11fa475e56e92f-7908edf8bbe8419ab21d0c93cd2a3b35 received from {username='slave'} at 127.0.0.1:33622
I20240502 14:01:12.071085 4167 tablet_copy_service.cc:433] P df655d53530449fc85de33de1c229815: ending tablet copy session b3988f2f38654eedac11fa475e56e92f-7908edf8bbe8419ab21d0c93cd2a3b35 on tablet 7908edf8bbe8419ab21d0c93cd2a3b35 with peer b3988f2f38654eedac11fa475e56e92f
I20240502 14:01:12.078193 4611 tablet_copy_client.cc:523] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:12.080425 4611 tablet_bootstrap.cc:492] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f: Bootstrap starting.
I20240502 14:01:12.100103 4611 tablet_bootstrap.cc:492] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=83 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:12.100386 4611 tablet_bootstrap.cc:492] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f: Bootstrap complete.
I20240502 14:01:12.100582 4611 ts_tablet_manager.cc:1367] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f: Time spent bootstrapping tablet: real 0.020s user 0.016s sys 0.004s
I20240502 14:01:12.101032 4611 raft_consensus.cc:348] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:12.101156 4611 raft_consensus.cc:729] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3988f2f38654eedac11fa475e56e92f, State: Initialized, Role: LEARNER
I20240502 14:01:12.101259 4611 consensus_queue.cc:260] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:12.101523 4611 ts_tablet_manager.cc:1398] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:12.101926 4016 tablet_copy_service.cc:342] P bd060dd4339f4fa5a985596bf7dec0ae: Request end of tablet copy session b3988f2f38654eedac11fa475e56e92f-b5172e93df3a4fb5a70188b0507639e6 received from {username='slave'} at 127.0.0.1:57094
I20240502 14:01:12.102002 4016 tablet_copy_service.cc:433] P bd060dd4339f4fa5a985596bf7dec0ae: ending tablet copy session b3988f2f38654eedac11fa475e56e92f-b5172e93df3a4fb5a70188b0507639e6 on tablet b5172e93df3a4fb5a70188b0507639e6 with peer b3988f2f38654eedac11fa475e56e92f
I20240502 14:01:12.127972 4616 tablet_copy_client.cc:523] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 5af426c57bc3434c9fbdcdff873a565c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:12.129194 4616 tablet_bootstrap.cc:492] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 5af426c57bc3434c9fbdcdff873a565c: Bootstrap starting.
I20240502 14:01:12.148618 4616 tablet_bootstrap.cc:492] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 5af426c57bc3434c9fbdcdff873a565c: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=101 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:12.148865 4616 tablet_bootstrap.cc:492] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 5af426c57bc3434c9fbdcdff873a565c: Bootstrap complete.
I20240502 14:01:12.149035 4616 ts_tablet_manager.cc:1367] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 5af426c57bc3434c9fbdcdff873a565c: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.007s
I20240502 14:01:12.149438 4616 raft_consensus.cc:348] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 5af426c57bc3434c9fbdcdff873a565c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } }
I20240502 14:01:12.149542 4616 raft_consensus.cc:729] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 5af426c57bc3434c9fbdcdff873a565c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5af426c57bc3434c9fbdcdff873a565c, State: Initialized, Role: LEARNER
I20240502 14:01:12.149626 4616 consensus_queue.cc:260] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 5af426c57bc3434c9fbdcdff873a565c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } }
I20240502 14:01:12.149937 4616 ts_tablet_manager.cc:1398] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 5af426c57bc3434c9fbdcdff873a565c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:12.150347 4092 tablet_copy_service.cc:342] P 24fd5d6c84c1483ea8f39bb975e26e45: Request end of tablet copy session 5af426c57bc3434c9fbdcdff873a565c-3d0d4aeaec264d04bae2c34e6edc6ec7 received from {username='slave'} at 127.0.0.1:59816
I20240502 14:01:12.150421 4092 tablet_copy_service.cc:433] P 24fd5d6c84c1483ea8f39bb975e26e45: ending tablet copy session 5af426c57bc3434c9fbdcdff873a565c-3d0d4aeaec264d04bae2c34e6edc6ec7 on tablet 3d0d4aeaec264d04bae2c34e6edc6ec7 with peer 5af426c57bc3434c9fbdcdff873a565c
I20240502 14:01:12.150624 4622 tablet_copy_client.cc:523] T a4d0e9d6d8e7445a9367e0f3a31c8748 P e1de31532912433dbed75c2bb01bc42a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:12.151773 4622 tablet_bootstrap.cc:492] T a4d0e9d6d8e7445a9367e0f3a31c8748 P e1de31532912433dbed75c2bb01bc42a: Bootstrap starting.
I20240502 14:01:12.171204 4622 tablet_bootstrap.cc:492] T a4d0e9d6d8e7445a9367e0f3a31c8748 P e1de31532912433dbed75c2bb01bc42a: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=98 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:12.171424 4622 tablet_bootstrap.cc:492] T a4d0e9d6d8e7445a9367e0f3a31c8748 P e1de31532912433dbed75c2bb01bc42a: Bootstrap complete.
I20240502 14:01:12.171574 4622 ts_tablet_manager.cc:1367] T a4d0e9d6d8e7445a9367e0f3a31c8748 P e1de31532912433dbed75c2bb01bc42a: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.004s
I20240502 14:01:12.171916 4622 raft_consensus.cc:348] T a4d0e9d6d8e7445a9367e0f3a31c8748 P e1de31532912433dbed75c2bb01bc42a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } }
I20240502 14:01:12.172009 4622 raft_consensus.cc:729] T a4d0e9d6d8e7445a9367e0f3a31c8748 P e1de31532912433dbed75c2bb01bc42a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e1de31532912433dbed75c2bb01bc42a, State: Initialized, Role: LEARNER
I20240502 14:01:12.172093 4622 consensus_queue.cc:260] T a4d0e9d6d8e7445a9367e0f3a31c8748 P e1de31532912433dbed75c2bb01bc42a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } }
I20240502 14:01:12.172396 4622 ts_tablet_manager.cc:1398] T a4d0e9d6d8e7445a9367e0f3a31c8748 P e1de31532912433dbed75c2bb01bc42a: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 14:01:12.172802 4092 tablet_copy_service.cc:342] P 24fd5d6c84c1483ea8f39bb975e26e45: Request end of tablet copy session e1de31532912433dbed75c2bb01bc42a-a4d0e9d6d8e7445a9367e0f3a31c8748 received from {username='slave'} at 127.0.0.1:59820
I20240502 14:01:12.172869 4092 tablet_copy_service.cc:433] P 24fd5d6c84c1483ea8f39bb975e26e45: ending tablet copy session e1de31532912433dbed75c2bb01bc42a-a4d0e9d6d8e7445a9367e0f3a31c8748 on tablet a4d0e9d6d8e7445a9367e0f3a31c8748 with peer e1de31532912433dbed75c2bb01bc42a
I20240502 14:01:12.205365 4478 raft_consensus.cc:1203] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f [term 2 LEARNER]: Deduplicated request from leader. Original: 2.27->[2.28-2.28] Dedup: 2.28->[]
I20240502 14:01:12.272927 4628 tablet_copy_client.cc:523] T dba8942501c440b18f71c7106878d0c4 P e1de31532912433dbed75c2bb01bc42a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:12.274488 4628 tablet_bootstrap.cc:492] T dba8942501c440b18f71c7106878d0c4 P e1de31532912433dbed75c2bb01bc42a: Bootstrap starting.
I20240502 14:01:12.281323 4633 tablet_copy_client.cc:523] T e6ca4998e3634ba789210e0c072e29b0 P 5af426c57bc3434c9fbdcdff873a565c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:12.282653 4633 tablet_bootstrap.cc:492] T e6ca4998e3634ba789210e0c072e29b0 P 5af426c57bc3434c9fbdcdff873a565c: Bootstrap starting.
I20240502 14:01:12.296614 4628 tablet_bootstrap.cc:492] T dba8942501c440b18f71c7106878d0c4 P e1de31532912433dbed75c2bb01bc42a: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=122 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:12.296939 4628 tablet_bootstrap.cc:492] T dba8942501c440b18f71c7106878d0c4 P e1de31532912433dbed75c2bb01bc42a: Bootstrap complete.
I20240502 14:01:12.297123 4628 ts_tablet_manager.cc:1367] T dba8942501c440b18f71c7106878d0c4 P e1de31532912433dbed75c2bb01bc42a: Time spent bootstrapping tablet: real 0.023s user 0.022s sys 0.000s
I20240502 14:01:12.297545 4628 raft_consensus.cc:348] T dba8942501c440b18f71c7106878d0c4 P e1de31532912433dbed75c2bb01bc42a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } }
I20240502 14:01:12.297669 4628 raft_consensus.cc:729] T dba8942501c440b18f71c7106878d0c4 P e1de31532912433dbed75c2bb01bc42a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e1de31532912433dbed75c2bb01bc42a, State: Initialized, Role: LEARNER
I20240502 14:01:12.297791 4628 consensus_queue.cc:260] T dba8942501c440b18f71c7106878d0c4 P e1de31532912433dbed75c2bb01bc42a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } }
I20240502 14:01:12.298415 4628 ts_tablet_manager.cc:1398] T dba8942501c440b18f71c7106878d0c4 P e1de31532912433dbed75c2bb01bc42a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:12.298786 4092 tablet_copy_service.cc:342] P 24fd5d6c84c1483ea8f39bb975e26e45: Request end of tablet copy session e1de31532912433dbed75c2bb01bc42a-dba8942501c440b18f71c7106878d0c4 received from {username='slave'} at 127.0.0.1:59820
I20240502 14:01:12.298851 4092 tablet_copy_service.cc:433] P 24fd5d6c84c1483ea8f39bb975e26e45: ending tablet copy session e1de31532912433dbed75c2bb01bc42a-dba8942501c440b18f71c7106878d0c4 on tablet dba8942501c440b18f71c7106878d0c4 with peer e1de31532912433dbed75c2bb01bc42a
I20240502 14:01:12.305830 4633 tablet_bootstrap.cc:492] T e6ca4998e3634ba789210e0c072e29b0 P 5af426c57bc3434c9fbdcdff873a565c: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=118 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:12.306082 4633 tablet_bootstrap.cc:492] T e6ca4998e3634ba789210e0c072e29b0 P 5af426c57bc3434c9fbdcdff873a565c: Bootstrap complete.
I20240502 14:01:12.306252 4633 ts_tablet_manager.cc:1367] T e6ca4998e3634ba789210e0c072e29b0 P 5af426c57bc3434c9fbdcdff873a565c: Time spent bootstrapping tablet: real 0.024s user 0.018s sys 0.004s
I20240502 14:01:12.306671 4633 raft_consensus.cc:348] T e6ca4998e3634ba789210e0c072e29b0 P 5af426c57bc3434c9fbdcdff873a565c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } }
I20240502 14:01:12.306792 4633 raft_consensus.cc:729] T e6ca4998e3634ba789210e0c072e29b0 P 5af426c57bc3434c9fbdcdff873a565c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5af426c57bc3434c9fbdcdff873a565c, State: Initialized, Role: LEARNER
I20240502 14:01:12.306919 4633 consensus_queue.cc:260] T e6ca4998e3634ba789210e0c072e29b0 P 5af426c57bc3434c9fbdcdff873a565c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } }
I20240502 14:01:12.307195 4633 ts_tablet_manager.cc:1398] T e6ca4998e3634ba789210e0c072e29b0 P 5af426c57bc3434c9fbdcdff873a565c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:01:12.307570 4016 tablet_copy_service.cc:342] P bd060dd4339f4fa5a985596bf7dec0ae: Request end of tablet copy session 5af426c57bc3434c9fbdcdff873a565c-e6ca4998e3634ba789210e0c072e29b0 received from {username='slave'} at 127.0.0.1:57102
I20240502 14:01:12.307647 4016 tablet_copy_service.cc:433] P bd060dd4339f4fa5a985596bf7dec0ae: ending tablet copy session 5af426c57bc3434c9fbdcdff873a565c-e6ca4998e3634ba789210e0c072e29b0 on tablet e6ca4998e3634ba789210e0c072e29b0 with peer 5af426c57bc3434c9fbdcdff873a565c
I20240502 14:01:12.402665 4401 raft_consensus.cc:1203] T e6ca4998e3634ba789210e0c072e29b0 P 5af426c57bc3434c9fbdcdff873a565c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20240502 14:01:12.452913 4478 raft_consensus.cc:3037] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [term 1 LEARNER]: Advancing to term 3
I20240502 14:01:12.454002 4478 raft_consensus.cc:1261] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [term 3 LEARNER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 14:01:12.454329 4672 consensus_queue.cc:1037] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:01:12.456225 4602 raft_consensus.cc:1050] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae: attempting to promote NON_VOTER b3988f2f38654eedac11fa475e56e92f to VOTER
I20240502 14:01:12.456638 4602 consensus_queue.cc:237] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 27, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
I20240502 14:01:12.457875 4154 raft_consensus.cc:1261] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815 [term 3 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 14:01:12.457955 4082 raft_consensus.cc:1261] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 14:01:12.458035 4478 raft_consensus.cc:1261] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [term 3 LEARNER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 14:01:12.458273 4679 consensus_queue.cc:1035] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:01:12.458495 4672 consensus_queue.cc:1035] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:01:12.458611 4678 consensus_queue.cc:1035] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:01:12.460171 4672 raft_consensus.cc:2932] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 LEADER]: Committing config change with OpId 3.29: config changed from index 26 to 29, b3988f2f38654eedac11fa475e56e92f (127.0.145.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:12.460829 4478 raft_consensus.cc:2932] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 26 to 29, b3988f2f38654eedac11fa475e56e92f (127.0.145.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:12.461990 4154 raft_consensus.cc:2932] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 26 to 29, b3988f2f38654eedac11fa475e56e92f (127.0.145.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:12.462179 3903 catalog_manager.cc:5484] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae reported cstate change: config changed from index 26 to 29, b3988f2f38654eedac11fa475e56e92f (127.0.145.69) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:01:12.462918 4082 raft_consensus.cc:2932] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 26 to 29, b3988f2f38654eedac11fa475e56e92f (127.0.145.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:12.466351 4000 consensus_queue.cc:237] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
I20240502 14:01:12.467439 4154 raft_consensus.cc:1261] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815 [term 3 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 14:01:12.467813 4478 raft_consensus.cc:1261] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [term 3 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 14:01:12.468153 4679 consensus_queue.cc:1035] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:01:12.468295 4678 consensus_queue.cc:1035] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:01:12.469830 4679 raft_consensus.cc:2932] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:12.470218 4478 raft_consensus.cc:2932] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:12.470269 4154 raft_consensus.cc:2932] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:12.471513 3889 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b5172e93df3a4fb5a70188b0507639e6 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:01:12.472047 3894 catalog_manager.cc:5484] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f reported cstate change: config changed from index 29 to 30, VOTER 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66) evicted. New cstate: current_term: 3 leader_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:12.474215 4067 tablet_service.cc:1514] Processing DeleteTablet for tablet b5172e93df3a4fb5a70188b0507639e6 with delete_type TABLET_DATA_TOMBSTONED (TS 24fd5d6c84c1483ea8f39bb975e26e45 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:59772
I20240502 14:01:12.474639 4684 tablet_replica.cc:330] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45: stopping tablet replica
I20240502 14:01:12.474768 4684 raft_consensus.cc:2229] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:12.475010 4684 raft_consensus.cc:2258] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:12.475597 4684 ts_tablet_manager.cc:1875] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:12.477206 4684 ts_tablet_manager.cc:1888] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.29
I20240502 14:01:12.477270 4684 log.cc:1196] T b5172e93df3a4fb5a70188b0507639e6 P 24fd5d6c84c1483ea8f39bb975e26e45: Deleting WAL directory at /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-1-root/wals/b5172e93df3a4fb5a70188b0507639e6
I20240502 14:01:12.477775 3890 catalog_manager.cc:4830] TS 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251): tablet b5172e93df3a4fb5a70188b0507639e6 (table test-workload [id=66735bdbe8fc4765af013422ce730ad0]) successfully deleted
I20240502 14:01:12.547775 4553 raft_consensus.cc:1203] T a4d0e9d6d8e7445a9367e0f3a31c8748 P e1de31532912433dbed75c2bb01bc42a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240502 14:01:12.583938 4553 raft_consensus.cc:1203] T dba8942501c440b18f71c7106878d0c4 P e1de31532912433dbed75c2bb01bc42a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20240502 14:01:12.606076 4686 raft_consensus.cc:1050] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815: attempting to promote NON_VOTER b3988f2f38654eedac11fa475e56e92f to VOTER
I20240502 14:01:12.606460 4686 consensus_queue.cc:237] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 22, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
I20240502 14:01:12.607736 4000 raft_consensus.cc:1261] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 14:01:12.607795 4478 raft_consensus.cc:1261] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f [term 2 LEARNER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 14:01:12.608012 4082 raft_consensus.cc:1261] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 14:01:12.608366 4688 consensus_queue.cc:1035] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:01:12.608566 4685 consensus_queue.cc:1035] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:01:12.608723 4664 consensus_queue.cc:1035] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:01:12.610280 4688 raft_consensus.cc:2932] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 28 to 29, b3988f2f38654eedac11fa475e56e92f (127.0.145.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:12.610651 4082 raft_consensus.cc:2932] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, b3988f2f38654eedac11fa475e56e92f (127.0.145.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:12.612795 3900 catalog_manager.cc:5484] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 reported cstate change: config changed from index 28 to 29, b3988f2f38654eedac11fa475e56e92f (127.0.145.69) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "df655d53530449fc85de33de1c229815" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:12.612874 4000 raft_consensus.cc:2932] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, b3988f2f38654eedac11fa475e56e92f (127.0.145.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:12.614074 4478 raft_consensus.cc:2932] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, b3988f2f38654eedac11fa475e56e92f (127.0.145.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:12.617319 4154 consensus_queue.cc:237] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 22, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
I20240502 14:01:12.618312 4082 raft_consensus.cc:1261] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 14:01:12.618520 4478 raft_consensus.cc:1261] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f [term 2 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 14:01:12.618824 4688 consensus_queue.cc:1035] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:01:12.618989 4686 consensus_queue.cc:1035] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:01:12.620193 4688 raft_consensus.cc:2932] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:12.620553 4478 raft_consensus.cc:2932] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:12.620914 4082 raft_consensus.cc:2932] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:12.621932 3890 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7908edf8bbe8419ab21d0c93cd2a3b35 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:01:12.622542 3898 catalog_manager.cc:5484] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 reported cstate change: config changed from index 29 to 30, VOTER bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65) evicted. New cstate: current_term: 2 leader_uuid: "df655d53530449fc85de33de1c229815" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:01:12.624487 3989 tablet_service.cc:1514] Processing DeleteTablet for tablet 7908edf8bbe8419ab21d0c93cd2a3b35 with delete_type TABLET_DATA_TOMBSTONED (TS bd060dd4339f4fa5a985596bf7dec0ae not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:57056
I20240502 14:01:12.624866 4692 tablet_replica.cc:330] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae: stopping tablet replica
I20240502 14:01:12.625025 4692 raft_consensus.cc:2229] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:12.625203 4692 raft_consensus.cc:2258] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:12.625774 4692 ts_tablet_manager.cc:1875] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:12.627535 4692 ts_tablet_manager.cc:1888] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.29
I20240502 14:01:12.627614 4692 log.cc:1196] T 7908edf8bbe8419ab21d0c93cd2a3b35 P bd060dd4339f4fa5a985596bf7dec0ae: Deleting WAL directory at /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-0-root/wals/7908edf8bbe8419ab21d0c93cd2a3b35
I20240502 14:01:12.628063 3889 catalog_manager.cc:4830] TS bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667): tablet 7908edf8bbe8419ab21d0c93cd2a3b35 (table test-workload [id=66735bdbe8fc4765af013422ce730ad0]) successfully deleted
I20240502 14:01:12.669596 3957 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:01:12.671309 4000 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b5172e93df3a4fb5a70188b0507639e6"
dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
mode: GRACEFUL
new_leader_uuid: "b3988f2f38654eedac11fa475e56e92f"
from {username='slave'} at 127.0.0.1:57080
I20240502 14:01:12.671450 4000 raft_consensus.cc:595] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 LEADER]: Received request to transfer leadership to TS b3988f2f38654eedac11fa475e56e92f
I20240502 14:01:12.672091 4000 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fb4c3dcde8764e17bb084cb37fef706f"
dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
mode: GRACEFUL
new_leader_uuid: "df655d53530449fc85de33de1c229815"
from {username='slave'} at 127.0.0.1:57080
I20240502 14:01:12.672194 4000 raft_consensus.cc:595] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [term 1 LEADER]: Received request to transfer leadership to TS df655d53530449fc85de33de1c229815
I20240502 14:01:12.672451 3957 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240502 14:01:12.672559 3957 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:01:12.721825 4401 raft_consensus.cc:1203] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 5af426c57bc3434c9fbdcdff873a565c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240502 14:01:12.786021 4602 raft_consensus.cc:1050] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae: attempting to promote NON_VOTER 5af426c57bc3434c9fbdcdff873a565c to VOTER
I20240502 14:01:12.786417 4602 consensus_queue.cc:237] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } }
I20240502 14:01:12.787627 4154 raft_consensus.cc:1261] T e6ca4998e3634ba789210e0c072e29b0 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 14:01:12.787627 4401 raft_consensus.cc:1261] T e6ca4998e3634ba789210e0c072e29b0 P 5af426c57bc3434c9fbdcdff873a565c [term 1 LEARNER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 14:01:12.788007 4078 raft_consensus.cc:1261] T e6ca4998e3634ba789210e0c072e29b0 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 14:01:12.787976 4672 consensus_queue.cc:1035] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:01:12.788164 4678 consensus_queue.cc:1035] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:01:12.788331 4602 consensus_queue.cc:1035] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:01:12.790230 4602 raft_consensus.cc:2932] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } }
I20240502 14:01:12.790560 4154 raft_consensus.cc:2932] T e6ca4998e3634ba789210e0c072e29b0 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } }
I20240502 14:01:12.790916 4401 raft_consensus.cc:2932] T e6ca4998e3634ba789210e0c072e29b0 P 5af426c57bc3434c9fbdcdff873a565c [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } }
I20240502 14:01:12.792785 3894 catalog_manager.cc:5484] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae reported cstate change: config changed from index 28 to 29, 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:01:12.793480 4082 raft_consensus.cc:2932] T e6ca4998e3634ba789210e0c072e29b0 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } }
I20240502 14:01:12.795388 4000 consensus_queue.cc:237] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } }
I20240502 14:01:12.796238 4401 raft_consensus.cc:1261] T e6ca4998e3634ba789210e0c072e29b0 P 5af426c57bc3434c9fbdcdff873a565c [term 1 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 14:01:12.796362 4154 raft_consensus.cc:1261] T e6ca4998e3634ba789210e0c072e29b0 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 14:01:12.796537 4678 consensus_queue.cc:1035] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:01:12.796676 4672 consensus_queue.cc:1035] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:01:12.798099 4672 raft_consensus.cc:2932] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } }
I20240502 14:01:12.799147 3889 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e6ca4998e3634ba789210e0c072e29b0 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:01:12.799523 4154 raft_consensus.cc:2932] T e6ca4998e3634ba789210e0c072e29b0 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } }
I20240502 14:01:12.799726 4401 raft_consensus.cc:2932] T e6ca4998e3634ba789210e0c072e29b0 P 5af426c57bc3434c9fbdcdff873a565c [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } }
I20240502 14:01:12.799922 3894 catalog_manager.cc:5484] T e6ca4998e3634ba789210e0c072e29b0 P bd060dd4339f4fa5a985596bf7dec0ae reported cstate change: config changed from index 29 to 30, VOTER 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66) evicted. New cstate: current_term: 1 leader_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:01:12.802378 4067 tablet_service.cc:1514] Processing DeleteTablet for tablet e6ca4998e3634ba789210e0c072e29b0 with delete_type TABLET_DATA_TOMBSTONED (TS 24fd5d6c84c1483ea8f39bb975e26e45 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:59772
I20240502 14:01:12.802565 4684 tablet_replica.cc:330] T e6ca4998e3634ba789210e0c072e29b0 P 24fd5d6c84c1483ea8f39bb975e26e45: stopping tablet replica
I20240502 14:01:12.802693 4684 raft_consensus.cc:2229] T e6ca4998e3634ba789210e0c072e29b0 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:12.802839 4684 raft_consensus.cc:2258] T e6ca4998e3634ba789210e0c072e29b0 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:12.803412 4684 ts_tablet_manager.cc:1875] T e6ca4998e3634ba789210e0c072e29b0 P 24fd5d6c84c1483ea8f39bb975e26e45: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:12.805096 4684 ts_tablet_manager.cc:1888] T e6ca4998e3634ba789210e0c072e29b0 P 24fd5d6c84c1483ea8f39bb975e26e45: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20240502 14:01:12.805157 4684 log.cc:1196] T e6ca4998e3634ba789210e0c072e29b0 P 24fd5d6c84c1483ea8f39bb975e26e45: Deleting WAL directory at /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-1-root/wals/e6ca4998e3634ba789210e0c072e29b0
I20240502 14:01:12.805598 3890 catalog_manager.cc:4830] TS 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251): tablet e6ca4998e3634ba789210e0c072e29b0 (table test-workload [id=66735bdbe8fc4765af013422ce730ad0]) successfully deleted
I20240502 14:01:12.976083 4698 raft_consensus.cc:1050] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45: attempting to promote NON_VOTER e1de31532912433dbed75c2bb01bc42a to VOTER
I20240502 14:01:12.976541 4698 consensus_queue.cc:237] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } }
I20240502 14:01:12.977936 4154 raft_consensus.cc:1261] T dba8942501c440b18f71c7106878d0c4 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 14:01:12.977937 4000 raft_consensus.cc:1261] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 14:01:12.978219 4553 raft_consensus.cc:1261] T dba8942501c440b18f71c7106878d0c4 P e1de31532912433dbed75c2bb01bc42a [term 1 LEARNER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 14:01:12.978250 4698 consensus_queue.cc:1035] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:01:12.978525 4640 consensus_queue.cc:1035] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:01:12.978691 4700 consensus_queue.cc:1035] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:01:12.980177 4698 raft_consensus.cc:2932] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, e1de31532912433dbed75c2bb01bc42a (127.0.145.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:12.980525 4154 raft_consensus.cc:2932] T dba8942501c440b18f71c7106878d0c4 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, e1de31532912433dbed75c2bb01bc42a (127.0.145.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:12.981254 4000 raft_consensus.cc:2932] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, e1de31532912433dbed75c2bb01bc42a (127.0.145.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:12.982381 4553 raft_consensus.cc:2932] T dba8942501c440b18f71c7106878d0c4 P e1de31532912433dbed75c2bb01bc42a [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, e1de31532912433dbed75c2bb01bc42a (127.0.145.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:12.983273 3900 catalog_manager.cc:5484] T dba8942501c440b18f71c7106878d0c4 P df655d53530449fc85de33de1c229815 reported cstate change: config changed from index 28 to 29, e1de31532912433dbed75c2bb01bc42a (127.0.145.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:12.986254 4078 consensus_queue.cc:237] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } }
I20240502 14:01:12.987217 4553 raft_consensus.cc:1261] T dba8942501c440b18f71c7106878d0c4 P e1de31532912433dbed75c2bb01bc42a [term 1 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 14:01:12.987399 4000 raft_consensus.cc:1261] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 14:01:12.987690 4678 raft_consensus.cc:979] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae: : Instructing follower b3988f2f38654eedac11fa475e56e92f to start an election
I20240502 14:01:12.987710 4640 consensus_queue.cc:1035] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:01:12.987777 4678 raft_consensus.cc:1067] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 LEADER]: Signalling peer b3988f2f38654eedac11fa475e56e92f to start an election
I20240502 14:01:12.987906 4700 consensus_queue.cc:1035] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:01:12.988128 4478 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b5172e93df3a4fb5a70188b0507639e6"
dest_uuid: "b3988f2f38654eedac11fa475e56e92f"
from {username='slave'} at 127.0.0.1:51162
I20240502 14:01:12.988237 4478 raft_consensus.cc:482] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:12.988299 4478 raft_consensus.cc:3037] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:01:12.989132 4478 raft_consensus.cc:504] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
I20240502 14:01:12.989187 4640 raft_consensus.cc:2932] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER df655d53530449fc85de33de1c229815 (127.0.145.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:12.989549 4553 raft_consensus.cc:2932] T dba8942501c440b18f71c7106878d0c4 P e1de31532912433dbed75c2bb01bc42a [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER df655d53530449fc85de33de1c229815 (127.0.145.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:12.989845 4154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5172e93df3a4fb5a70188b0507639e6" candidate_uuid: "b3988f2f38654eedac11fa475e56e92f" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "df655d53530449fc85de33de1c229815"
I20240502 14:01:12.989935 4154 raft_consensus.cc:3037] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:01:12.990676 4154 raft_consensus.cc:2454] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3988f2f38654eedac11fa475e56e92f in term 4.
I20240502 14:01:12.990960 4443 leader_election.cc:302] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3988f2f38654eedac11fa475e56e92f, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:12.991012 4005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5172e93df3a4fb5a70188b0507639e6" candidate_uuid: "b3988f2f38654eedac11fa475e56e92f" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
I20240502 14:01:12.991089 4005 raft_consensus.cc:3032] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:01:12.991123 4005 raft_consensus.cc:729] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Running, Role: LEADER
I20240502 14:01:12.991221 3890 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dba8942501c440b18f71c7106878d0c4 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:01:12.991235 4005 consensus_queue.cc:260] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
I20240502 14:01:12.991401 4005 raft_consensus.cc:3037] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:01:12.991758 4666 raft_consensus.cc:2781] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:01:12.991818 3894 catalog_manager.cc:5484] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 reported cstate change: config changed from index 29 to 30, VOTER df655d53530449fc85de33de1c229815 (127.0.145.67) evicted. New cstate: current_term: 1 leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:01:12.991926 4666 raft_consensus.cc:686] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [term 4 LEADER]: Becoming Leader. State: Replica: b3988f2f38654eedac11fa475e56e92f, State: Running, Role: LEADER
I20240502 14:01:12.992048 4666 consensus_queue.cc:237] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
I20240502 14:01:12.992256 4005 raft_consensus.cc:2454] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3988f2f38654eedac11fa475e56e92f in term 4.
I20240502 14:01:12.989559 4478 leader_election.cc:288] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [CANDIDATE]: Term 4 election: Requested vote from peers bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:12.993429 3898 catalog_manager.cc:5484] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f reported cstate change: term changed from 3 to 4, leader changed from bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65) to b3988f2f38654eedac11fa475e56e92f (127.0.145.69). New cstate: current_term: 4 leader_uuid: "b3988f2f38654eedac11fa475e56e92f" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:01:12.994150 4000 raft_consensus.cc:2932] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER df655d53530449fc85de33de1c229815 (127.0.145.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:12.995218 4142 tablet_service.cc:1514] Processing DeleteTablet for tablet dba8942501c440b18f71c7106878d0c4 with delete_type TABLET_DATA_TOMBSTONED (TS df655d53530449fc85de33de1c229815 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:33568
I20240502 14:01:12.996901 4705 tablet_replica.cc:330] T dba8942501c440b18f71c7106878d0c4 P df655d53530449fc85de33de1c229815: stopping tablet replica
I20240502 14:01:12.997031 4705 raft_consensus.cc:2229] T dba8942501c440b18f71c7106878d0c4 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:12.997156 4705 raft_consensus.cc:2258] T dba8942501c440b18f71c7106878d0c4 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:12.997664 4705 ts_tablet_manager.cc:1875] T dba8942501c440b18f71c7106878d0c4 P df655d53530449fc85de33de1c229815: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:12.999358 4705 ts_tablet_manager.cc:1888] T dba8942501c440b18f71c7106878d0c4 P df655d53530449fc85de33de1c229815: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20240502 14:01:12.999435 4705 log.cc:1196] T dba8942501c440b18f71c7106878d0c4 P df655d53530449fc85de33de1c229815: Deleting WAL directory at /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-2-root/wals/dba8942501c440b18f71c7106878d0c4
I20240502 14:01:12.999917 3890 catalog_manager.cc:4830] TS df655d53530449fc85de33de1c229815 (127.0.145.67:34309): tablet dba8942501c440b18f71c7106878d0c4 (table test-workload [id=66735bdbe8fc4765af013422ce730ad0]) successfully deleted
I20240502 14:01:13.074182 4602 raft_consensus.cc:979] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae: : Instructing follower df655d53530449fc85de33de1c229815 to start an election
I20240502 14:01:13.074298 4678 raft_consensus.cc:1067] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [term 1 LEADER]: Signalling peer df655d53530449fc85de33de1c229815 to start an election
I20240502 14:01:13.074677 4154 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fb4c3dcde8764e17bb084cb37fef706f"
dest_uuid: "df655d53530449fc85de33de1c229815"
from {username='slave'} at 127.0.0.1:33594
I20240502 14:01:13.074815 4154 raft_consensus.cc:482] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:13.074882 4154 raft_consensus.cc:3037] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:13.075747 4154 raft_consensus.cc:504] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } }
I20240502 14:01:13.076095 4154 leader_election.cc:288] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [CANDIDATE]: Term 2 election: Requested vote from peers 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251), bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667)
I20240502 14:01:13.076333 4078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb4c3dcde8764e17bb084cb37fef706f" candidate_uuid: "df655d53530449fc85de33de1c229815" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
I20240502 14:01:13.076432 4000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb4c3dcde8764e17bb084cb37fef706f" candidate_uuid: "df655d53530449fc85de33de1c229815" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
I20240502 14:01:13.076442 4078 raft_consensus.cc:3037] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:13.076535 4000 raft_consensus.cc:3032] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:01:13.076575 4000 raft_consensus.cc:729] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Running, Role: LEADER
I20240502 14:01:13.076701 4000 consensus_queue.cc:260] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } }
I20240502 14:01:13.076881 4000 raft_consensus.cc:3037] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:13.077420 4078 raft_consensus.cc:2454] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df655d53530449fc85de33de1c229815 in term 2.
I20240502 14:01:13.077713 4123 leader_election.cc:302] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:13.077735 4000 raft_consensus.cc:2454] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df655d53530449fc85de33de1c229815 in term 2.
I20240502 14:01:13.077885 4686 raft_consensus.cc:2781] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:13.077960 4686 raft_consensus.cc:686] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [term 2 LEADER]: Becoming Leader. State: Replica: df655d53530449fc85de33de1c229815, State: Running, Role: LEADER
I20240502 14:01:13.078071 4686 consensus_queue.cc:237] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } }
I20240502 14:01:13.079463 3898 catalog_manager.cc:5484] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 reported cstate change: term changed from 1 to 2, leader changed from bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65) to df655d53530449fc85de33de1c229815 (127.0.145.67). New cstate: current_term: 2 leader_uuid: "df655d53530449fc85de33de1c229815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:13.112938 4698 raft_consensus.cc:1050] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45: attempting to promote NON_VOTER e1de31532912433dbed75c2bb01bc42a to VOTER
I20240502 14:01:13.113304 4698 consensus_queue.cc:237] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } }
I20240502 14:01:13.114548 4553 raft_consensus.cc:1261] T a4d0e9d6d8e7445a9367e0f3a31c8748 P e1de31532912433dbed75c2bb01bc42a [term 1 LEARNER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 14:01:13.114929 4000 raft_consensus.cc:1261] T a4d0e9d6d8e7445a9367e0f3a31c8748 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 14:01:13.114905 4697 consensus_queue.cc:1035] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:01:13.115547 4697 consensus_queue.cc:1035] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:01:13.115859 4154 raft_consensus.cc:1261] T a4d0e9d6d8e7445a9367e0f3a31c8748 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 14:01:13.116173 4697 consensus_queue.cc:1035] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:01:13.117436 4698 raft_consensus.cc:2932] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, e1de31532912433dbed75c2bb01bc42a (127.0.145.70) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:13.117785 4000 raft_consensus.cc:2932] T a4d0e9d6d8e7445a9367e0f3a31c8748 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, e1de31532912433dbed75c2bb01bc42a (127.0.145.70) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:13.119555 4553 raft_consensus.cc:2932] T a4d0e9d6d8e7445a9367e0f3a31c8748 P e1de31532912433dbed75c2bb01bc42a [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, e1de31532912433dbed75c2bb01bc42a (127.0.145.70) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:13.119711 3898 catalog_manager.cc:5484] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 reported cstate change: config changed from index 27 to 28, e1de31532912433dbed75c2bb01bc42a (127.0.145.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:01:13.120333 4154 raft_consensus.cc:2932] T a4d0e9d6d8e7445a9367e0f3a31c8748 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, e1de31532912433dbed75c2bb01bc42a (127.0.145.70) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:13.122745 4078 consensus_queue.cc:237] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } }
I20240502 14:01:13.123684 4553 raft_consensus.cc:1261] T a4d0e9d6d8e7445a9367e0f3a31c8748 P e1de31532912433dbed75c2bb01bc42a [term 1 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 14:01:13.123958 4640 consensus_queue.cc:1035] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:01:13.124006 4000 raft_consensus.cc:1261] T a4d0e9d6d8e7445a9367e0f3a31c8748 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 14:01:13.124413 4640 consensus_queue.cc:1035] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:01:13.125142 4697 raft_consensus.cc:2932] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER df655d53530449fc85de33de1c229815 (127.0.145.67) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:13.125571 4553 raft_consensus.cc:2932] T a4d0e9d6d8e7445a9367e0f3a31c8748 P e1de31532912433dbed75c2bb01bc42a [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER df655d53530449fc85de33de1c229815 (127.0.145.67) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:13.127043 3890 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a4d0e9d6d8e7445a9367e0f3a31c8748 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:01:13.127271 3898 catalog_manager.cc:5484] T a4d0e9d6d8e7445a9367e0f3a31c8748 P e1de31532912433dbed75c2bb01bc42a reported cstate change: config changed from index 28 to 29, VOTER df655d53530449fc85de33de1c229815 (127.0.145.67) evicted. New cstate: current_term: 1 leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:13.127761 4000 raft_consensus.cc:2932] T a4d0e9d6d8e7445a9367e0f3a31c8748 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER df655d53530449fc85de33de1c229815 (127.0.145.67) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:13.129412 4142 tablet_service.cc:1514] Processing DeleteTablet for tablet a4d0e9d6d8e7445a9367e0f3a31c8748 with delete_type TABLET_DATA_TOMBSTONED (TS df655d53530449fc85de33de1c229815 not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:33568
I20240502 14:01:13.129585 4705 tablet_replica.cc:330] T a4d0e9d6d8e7445a9367e0f3a31c8748 P df655d53530449fc85de33de1c229815: stopping tablet replica
I20240502 14:01:13.129720 4705 raft_consensus.cc:2229] T a4d0e9d6d8e7445a9367e0f3a31c8748 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:13.129858 4705 raft_consensus.cc:2258] T a4d0e9d6d8e7445a9367e0f3a31c8748 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:13.130383 4705 ts_tablet_manager.cc:1875] T a4d0e9d6d8e7445a9367e0f3a31c8748 P df655d53530449fc85de33de1c229815: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:13.132138 4705 ts_tablet_manager.cc:1888] T a4d0e9d6d8e7445a9367e0f3a31c8748 P df655d53530449fc85de33de1c229815: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20240502 14:01:13.132210 4705 log.cc:1196] T a4d0e9d6d8e7445a9367e0f3a31c8748 P df655d53530449fc85de33de1c229815: Deleting WAL directory at /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-2-root/wals/a4d0e9d6d8e7445a9367e0f3a31c8748
I20240502 14:01:13.132630 3890 catalog_manager.cc:4830] TS df655d53530449fc85de33de1c229815 (127.0.145.67:34309): tablet a4d0e9d6d8e7445a9367e0f3a31c8748 (table test-workload [id=66735bdbe8fc4765af013422ce730ad0]) successfully deleted
I20240502 14:01:13.244526 4700 raft_consensus.cc:1050] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45: attempting to promote NON_VOTER 5af426c57bc3434c9fbdcdff873a565c to VOTER
I20240502 14:01:13.244972 4700 consensus_queue.cc:237] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } }
I20240502 14:01:13.246191 4401 raft_consensus.cc:1261] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 5af426c57bc3434c9fbdcdff873a565c [term 1 LEARNER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 14:01:13.246224 4154 raft_consensus.cc:1261] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 14:01:13.246472 4000 raft_consensus.cc:1261] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 14:01:13.246522 4640 consensus_queue.cc:1035] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:01:13.246821 4640 consensus_queue.cc:1035] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:01:13.247090 4640 consensus_queue.cc:1035] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:01:13.248438 4697 raft_consensus.cc:2932] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } }
I20240502 14:01:13.248803 4000 raft_consensus.cc:2932] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } }
I20240502 14:01:13.249394 4401 raft_consensus.cc:2932] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 5af426c57bc3434c9fbdcdff873a565c [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } }
I20240502 14:01:13.250375 4154 raft_consensus.cc:2932] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } }
I20240502 14:01:13.251667 3898 catalog_manager.cc:5484] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 reported cstate change: config changed from index 27 to 28, 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:01:13.254694 4078 consensus_queue.cc:237] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } }
I20240502 14:01:13.255584 4154 raft_consensus.cc:1261] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 14:01:13.255645 4401 raft_consensus.cc:1261] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 5af426c57bc3434c9fbdcdff873a565c [term 1 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 14:01:13.255957 4698 consensus_queue.cc:1035] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:01:13.256114 4700 consensus_queue.cc:1035] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:01:13.257499 4700 raft_consensus.cc:2932] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } }
I20240502 14:01:13.257925 4154 raft_consensus.cc:2932] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } }
I20240502 14:01:13.258100 4401 raft_consensus.cc:2932] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 5af426c57bc3434c9fbdcdff873a565c [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } }
I20240502 14:01:13.258754 3890 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3d0d4aeaec264d04bae2c34e6edc6ec7 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:01:13.259716 3900 catalog_manager.cc:5484] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P df655d53530449fc85de33de1c229815 reported cstate change: config changed from index 28 to 29, VOTER bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65) evicted. New cstate: current_term: 1 leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } }
I20240502 14:01:13.261704 3989 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d0d4aeaec264d04bae2c34e6edc6ec7 with delete_type TABLET_DATA_TOMBSTONED (TS bd060dd4339f4fa5a985596bf7dec0ae not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:57056
I20240502 14:01:13.262039 4715 tablet_replica.cc:330] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P bd060dd4339f4fa5a985596bf7dec0ae: stopping tablet replica
I20240502 14:01:13.262158 4715 raft_consensus.cc:2229] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:13.262287 4715 raft_consensus.cc:2258] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:13.262753 4715 ts_tablet_manager.cc:1875] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P bd060dd4339f4fa5a985596bf7dec0ae: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:13.264245 4715 ts_tablet_manager.cc:1888] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P bd060dd4339f4fa5a985596bf7dec0ae: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20240502 14:01:13.264307 4715 log.cc:1196] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P bd060dd4339f4fa5a985596bf7dec0ae: Deleting WAL directory at /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-0-root/wals/3d0d4aeaec264d04bae2c34e6edc6ec7
I20240502 14:01:13.264672 3889 catalog_manager.cc:4830] TS bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667): tablet 3d0d4aeaec264d04bae2c34e6edc6ec7 (table test-workload [id=66735bdbe8fc4765af013422ce730ad0]) successfully deleted
I20240502 14:01:13.382942 4154 raft_consensus.cc:1261] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815 [term 4 FOLLOWER]: Refusing update from remote peer b3988f2f38654eedac11fa475e56e92f: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240502 14:01:13.383323 4666 consensus_queue.cc:1035] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:01:13.385813 4000 raft_consensus.cc:1261] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 4 FOLLOWER]: Refusing update from remote peer b3988f2f38654eedac11fa475e56e92f: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240502 14:01:13.386129 4666 consensus_queue.cc:1035] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:01:13.493330 4078 raft_consensus.cc:1261] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 14:01:13.493695 4686 consensus_queue.cc:1035] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:01:13.495699 4000 raft_consensus.cc:1261] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 14:01:13.496107 4688 consensus_queue.cc:1035] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:01:14.263533 4078 consensus_queue.cc:237] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } }
I20240502 14:01:14.264684 4000 raft_consensus.cc:1261] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 14:01:14.264755 4154 raft_consensus.cc:1261] T 46b4a50dee854f62aac3a02889525b40 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
W20240502 14:01:14.264879 4045 consensus_peers.cc:485] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 -> Peer e1de31532912433dbed75c2bb01bc42a (127.0.145.70:41637): Couldn't send request to peer e1de31532912433dbed75c2bb01bc42a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 46b4a50dee854f62aac3a02889525b40. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:14.265043 4725 consensus_queue.cc:1035] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:01:14.265206 4727 consensus_queue.cc:1035] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:01:14.266620 4727 raft_consensus.cc:2932] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER e1de31532912433dbed75c2bb01bc42a (127.0.145.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } } }
I20240502 14:01:14.267024 4000 raft_consensus.cc:2932] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER e1de31532912433dbed75c2bb01bc42a (127.0.145.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } } }
I20240502 14:01:14.267163 4154 raft_consensus.cc:2932] T 46b4a50dee854f62aac3a02889525b40 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER e1de31532912433dbed75c2bb01bc42a (127.0.145.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } } }
I20240502 14:01:14.269023 3894 catalog_manager.cc:5484] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 reported cstate change: config changed from index -1 to 28, NON_VOTER e1de31532912433dbed75c2bb01bc42a (127.0.145.70) added. New cstate: current_term: 1 leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:14.269680 4000 consensus_queue.cc:237] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:14.271003 4078 raft_consensus.cc:1261] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 14:01:14.271045 4154 raft_consensus.cc:1261] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 2 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
W20240502 14:01:14.271225 3969 consensus_peers.cc:485] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae -> Peer b3988f2f38654eedac11fa475e56e92f (127.0.145.69:45021): Couldn't send request to peer b3988f2f38654eedac11fa475e56e92f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 144739b71bf6474d90399816120b3b59. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:14.271353 4733 consensus_queue.cc:1035] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:01:14.271497 4734 consensus_queue.cc:1035] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:01:14.273202 4734 raft_consensus.cc:2932] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER b3988f2f38654eedac11fa475e56e92f (127.0.145.69) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } } }
I20240502 14:01:14.273551 4078 raft_consensus.cc:2932] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER b3988f2f38654eedac11fa475e56e92f (127.0.145.69) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } } }
I20240502 14:01:14.275159 4150 raft_consensus.cc:2932] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER b3988f2f38654eedac11fa475e56e92f (127.0.145.69) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } } }
I20240502 14:01:14.275285 4154 consensus_queue.cc:237] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } }
I20240502 14:01:14.275399 3900 catalog_manager.cc:5484] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 reported cstate change: config changed from index -1 to 29, NON_VOTER b3988f2f38654eedac11fa475e56e92f (127.0.145.69) added. New cstate: current_term: 2 leader_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } } }
I20240502 14:01:14.277210 4000 raft_consensus.cc:1261] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 14:01:14.277715 4078 raft_consensus.cc:1261] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 14:01:14.277779 4740 consensus_queue.cc:1035] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:01:14.278008 4688 consensus_queue.cc:1035] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:01:14.279999 4688 raft_consensus.cc:2932] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } } }
I20240502 14:01:14.280989 4000 raft_consensus.cc:2932] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } } }
W20240502 14:01:14.281477 4123 consensus_peers.cc:485] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 -> Peer 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68:34601): Couldn't send request to peer 5af426c57bc3434c9fbdcdff873a565c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ac79a5b1f514bdcb36e0db4ff6e3a59. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:14.281479 4078 raft_consensus.cc:2932] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } } }
I20240502 14:01:14.282946 3894 catalog_manager.cc:5484] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 reported cstate change: config changed from index -1 to 28, NON_VOTER 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) added. New cstate: current_term: 1 leader_uuid: "df655d53530449fc85de33de1c229815" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:14.286250 4478 consensus_queue.cc:237] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } }
I20240502 14:01:14.287340 4000 raft_consensus.cc:1261] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 4 FOLLOWER]: Refusing update from remote peer b3988f2f38654eedac11fa475e56e92f: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240502 14:01:14.287588 4150 raft_consensus.cc:1261] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815 [term 4 FOLLOWER]: Refusing update from remote peer b3988f2f38654eedac11fa475e56e92f: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240502 14:01:14.287616 4666 consensus_queue.cc:1035] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 14:01:14.287890 4666 consensus_queue.cc:1035] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 14:01:14.289356 4722 raft_consensus.cc:2932] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [term 4 LEADER]: Committing config change with OpId 4.32: config changed from index 30 to 32, NON_VOTER 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } } }
I20240502 14:01:14.289960 4150 raft_consensus.cc:2932] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 30 to 32, NON_VOTER 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } } }
I20240502 14:01:14.291103 3900 catalog_manager.cc:5484] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f reported cstate change: config changed from index 30 to 32, NON_VOTER 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) added. New cstate: current_term: 4 leader_uuid: "b3988f2f38654eedac11fa475e56e92f" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:14.291355 4000 raft_consensus.cc:2932] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 30 to 32, NON_VOTER 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } } }
W20240502 14:01:14.291613 4443 consensus_peers.cc:485] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f -> Peer 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68:34601): Couldn't send request to peer 5af426c57bc3434c9fbdcdff873a565c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5172e93df3a4fb5a70188b0507639e6. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:14.293148 4150 consensus_queue.cc:237] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 23, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
W20240502 14:01:14.294468 4120 consensus_peers.cc:485] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 -> Peer b3988f2f38654eedac11fa475e56e92f (127.0.145.69:45021): Couldn't send request to peer b3988f2f38654eedac11fa475e56e92f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c965f0e11bc1433f8ccdc20c98b4b4b9. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:14.294653 4000 raft_consensus.cc:1261] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 14:01:14.294687 4078 raft_consensus.cc:1261] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 14:01:14.294946 4724 consensus_queue.cc:1035] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:01:14.295091 4688 consensus_queue.cc:1035] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:01:14.296597 4724 raft_consensus.cc:2932] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER b3988f2f38654eedac11fa475e56e92f (127.0.145.69) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } } }
I20240502 14:01:14.296958 4078 raft_consensus.cc:2932] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER b3988f2f38654eedac11fa475e56e92f (127.0.145.69) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } } }
I20240502 14:01:14.297080 4000 raft_consensus.cc:2932] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER b3988f2f38654eedac11fa475e56e92f (127.0.145.69) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } } }
I20240502 14:01:14.298857 4150 consensus_queue.cc:237] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } }
I20240502 14:01:14.298954 3903 catalog_manager.cc:5484] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 reported cstate change: config changed from index -1 to 29, NON_VOTER b3988f2f38654eedac11fa475e56e92f (127.0.145.69) added. New cstate: current_term: 2 leader_uuid: "df655d53530449fc85de33de1c229815" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:14.300240 4000 raft_consensus.cc:1261] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 14:01:14.300668 4078 raft_consensus.cc:1261] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 14:01:14.300704 4688 consensus_queue.cc:1035] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:01:14.301024 4688 consensus_queue.cc:1035] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:01:14.302347 4740 raft_consensus.cc:2932] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER e1de31532912433dbed75c2bb01bc42a (127.0.145.70) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } } }
I20240502 14:01:14.302695 4078 raft_consensus.cc:2932] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER e1de31532912433dbed75c2bb01bc42a (127.0.145.70) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } } }
W20240502 14:01:14.304107 4120 consensus_peers.cc:485] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 -> Peer e1de31532912433dbed75c2bb01bc42a (127.0.145.70:41637): Couldn't send request to peer e1de31532912433dbed75c2bb01bc42a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb4c3dcde8764e17bb084cb37fef706f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:14.304636 3900 catalog_manager.cc:5484] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45 reported cstate change: config changed from index -1 to 29, NON_VOTER e1de31532912433dbed75c2bb01bc42a (127.0.145.70) added. New cstate: current_term: 2 leader_uuid: "df655d53530449fc85de33de1c229815" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } } }
I20240502 14:01:14.304893 4000 raft_consensus.cc:2932] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER e1de31532912433dbed75c2bb01bc42a (127.0.145.70) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } } }
I20240502 14:01:14.671846 4752 ts_tablet_manager.cc:902] T c965f0e11bc1433f8ccdc20c98b4b4b9 P b3988f2f38654eedac11fa475e56e92f: Initiating tablet copy from peer df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:14.672444 4752 tablet_copy_client.cc:308] T c965f0e11bc1433f8ccdc20c98b4b4b9 P b3988f2f38654eedac11fa475e56e92f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.67:34309
I20240502 14:01:14.672796 3957 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:01:14.673010 4167 tablet_copy_service.cc:140] P df655d53530449fc85de33de1c229815: Received BeginTabletCopySession request for tablet c965f0e11bc1433f8ccdc20c98b4b4b9 from peer b3988f2f38654eedac11fa475e56e92f ({username='slave'} at 127.0.0.1:33622)
I20240502 14:01:14.673107 4167 tablet_copy_service.cc:161] P df655d53530449fc85de33de1c229815: Beginning new tablet copy session on tablet c965f0e11bc1433f8ccdc20c98b4b4b9 from peer b3988f2f38654eedac11fa475e56e92f at {username='slave'} at 127.0.0.1:33622: session id = b3988f2f38654eedac11fa475e56e92f-c965f0e11bc1433f8ccdc20c98b4b4b9
I20240502 14:01:14.674141 4167 tablet_copy_source_session.cc:205] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:14.674638 4000 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "144739b71bf6474d90399816120b3b59"
dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
mode: GRACEFUL
new_leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
from {username='slave'} at 127.0.0.1:57080
I20240502 14:01:14.674765 4000 raft_consensus.cc:595] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 LEADER]: Received request to transfer leadership to TS 24fd5d6c84c1483ea8f39bb975e26e45
I20240502 14:01:14.675343 4752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c965f0e11bc1433f8ccdc20c98b4b4b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:14.675451 4078 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "46b4a50dee854f62aac3a02889525b40"
dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
mode: GRACEFUL
new_leader_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
from {username='slave'} at 127.0.0.1:59812
I20240502 14:01:14.675568 4078 raft_consensus.cc:595] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Received request to transfer leadership to TS bd060dd4339f4fa5a985596bf7dec0ae
I20240502 14:01:14.676434 4150 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7908edf8bbe8419ab21d0c93cd2a3b35"
dest_uuid: "df655d53530449fc85de33de1c229815"
mode: GRACEFUL
new_leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
from {username='slave'} at 127.0.0.1:33630
I20240502 14:01:14.676551 4150 raft_consensus.cc:595] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [term 2 LEADER]: Received request to transfer leadership to TS 24fd5d6c84c1483ea8f39bb975e26e45
I20240502 14:01:14.677186 4078 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a4d0e9d6d8e7445a9367e0f3a31c8748"
dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
mode: GRACEFUL
new_leader_uuid: "e1de31532912433dbed75c2bb01bc42a"
from {username='slave'} at 127.0.0.1:59812
I20240502 14:01:14.677295 4078 raft_consensus.cc:595] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Received request to transfer leadership to TS e1de31532912433dbed75c2bb01bc42a
I20240502 14:01:14.678050 4752 tablet_copy_client.cc:783] T c965f0e11bc1433f8ccdc20c98b4b4b9 P b3988f2f38654eedac11fa475e56e92f: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:14.678200 4752 tablet_copy_client.cc:647] T c965f0e11bc1433f8ccdc20c98b4b4b9 P b3988f2f38654eedac11fa475e56e92f: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:14.678179 4150 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c965f0e11bc1433f8ccdc20c98b4b4b9"
dest_uuid: "df655d53530449fc85de33de1c229815"
mode: GRACEFUL
new_leader_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
from {username='slave'} at 127.0.0.1:33630
I20240502 14:01:14.678323 4150 raft_consensus.cc:595] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 2 LEADER]: Received request to transfer leadership to TS bd060dd4339f4fa5a985596bf7dec0ae
I20240502 14:01:14.678618 3957 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240502 14:01:14.678742 3957 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:01:14.679199 4754 tablet_copy_client.cc:1002] T c965f0e11bc1433f8ccdc20c98b4b4b9 P b3988f2f38654eedac11fa475e56e92f: tablet copy: Injecting latency into file download: 2000
I20240502 14:01:14.688130 4724 raft_consensus.cc:979] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815: : Instructing follower 24fd5d6c84c1483ea8f39bb975e26e45 to start an election
I20240502 14:01:14.688228 4724 raft_consensus.cc:1067] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [term 2 LEADER]: Signalling peer 24fd5d6c84c1483ea8f39bb975e26e45 to start an election
I20240502 14:01:14.688597 4078 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7908edf8bbe8419ab21d0c93cd2a3b35"
dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
from {username='slave'} at 127.0.0.1:59788
I20240502 14:01:14.688714 4078 raft_consensus.cc:482] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:14.688803 4078 raft_consensus.cc:3037] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:01:14.689680 4078 raft_consensus.cc:504] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
I20240502 14:01:14.690025 4078 leader_election.cc:288] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 3 election: Requested vote from peers df655d53530449fc85de33de1c229815 (127.0.145.67:34309), b3988f2f38654eedac11fa475e56e92f (127.0.145.69:45021)
I20240502 14:01:14.690229 4154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7908edf8bbe8419ab21d0c93cd2a3b35" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "df655d53530449fc85de33de1c229815"
I20240502 14:01:14.690349 4154 raft_consensus.cc:3032] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:01:14.690392 4154 raft_consensus.cc:729] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df655d53530449fc85de33de1c229815, State: Running, Role: LEADER
I20240502 14:01:14.690517 4154 consensus_queue.cc:260] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
I20240502 14:01:14.690662 4154 raft_consensus.cc:3037] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:01:14.691450 4154 raft_consensus.cc:2454] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 3.
I20240502 14:01:14.691732 4048 leader_election.cc:302] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:14.691744 4478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7908edf8bbe8419ab21d0c93cd2a3b35" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "b3988f2f38654eedac11fa475e56e92f"
I20240502 14:01:14.691879 4478 raft_consensus.cc:3037] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:01:14.691884 4700 raft_consensus.cc:2781] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:01:14.692011 4700 raft_consensus.cc:686] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 LEADER]: Becoming Leader. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Running, Role: LEADER
I20240502 14:01:14.692126 4700 consensus_queue.cc:237] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
I20240502 14:01:14.692682 4478 raft_consensus.cc:2454] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 3.
I20240502 14:01:14.693796 3894 catalog_manager.cc:5484] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 reported cstate change: term changed from 2 to 3, leader changed from df655d53530449fc85de33de1c229815 (127.0.145.67) to 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66). New cstate: current_term: 3 leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:14.705811 4724 raft_consensus.cc:979] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815: : Instructing follower bd060dd4339f4fa5a985596bf7dec0ae to start an election
I20240502 14:01:14.705917 4724 raft_consensus.cc:1067] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 2 LEADER]: Signalling peer bd060dd4339f4fa5a985596bf7dec0ae to start an election
I20240502 14:01:14.706235 4000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c965f0e11bc1433f8ccdc20c98b4b4b9"
dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
from {username='slave'} at 127.0.0.1:57070
I20240502 14:01:14.706348 4000 raft_consensus.cc:482] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:14.706400 4000 raft_consensus.cc:3037] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:01:14.707147 4000 raft_consensus.cc:504] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:14.707445 4000 leader_election.cc:288] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 3 election: Requested vote from peers 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:14.707660 4078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c965f0e11bc1433f8ccdc20c98b4b4b9" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
I20240502 14:01:14.707743 4150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c965f0e11bc1433f8ccdc20c98b4b4b9" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "df655d53530449fc85de33de1c229815"
I20240502 14:01:14.707762 4078 raft_consensus.cc:3037] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:01:14.707828 4150 raft_consensus.cc:3032] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:01:14.707863 4150 raft_consensus.cc:729] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df655d53530449fc85de33de1c229815, State: Running, Role: LEADER
I20240502 14:01:14.707984 4150 consensus_queue.cc:260] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:14.708177 4150 raft_consensus.cc:3037] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:01:14.708659 4078 raft_consensus.cc:2454] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 3.
I20240502 14:01:14.708896 4150 raft_consensus.cc:2454] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 3.
I20240502 14:01:14.708945 3972 leader_election.cc:302] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, bd060dd4339f4fa5a985596bf7dec0ae; no voters:
I20240502 14:01:14.709112 4678 raft_consensus.cc:2781] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:01:14.709196 4678 raft_consensus.cc:686] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 LEADER]: Becoming Leader. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Running, Role: LEADER
I20240502 14:01:14.709311 4678 consensus_queue.cc:237] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:14.710723 3894 catalog_manager.cc:5484] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae reported cstate change: term changed from 2 to 3, leader changed from df655d53530449fc85de33de1c229815 (127.0.145.67) to bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65). New cstate: current_term: 3 leader_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:14.714794 4758 ts_tablet_manager.cc:902] T b5172e93df3a4fb5a70188b0507639e6 P 5af426c57bc3434c9fbdcdff873a565c: Initiating tablet copy from peer b3988f2f38654eedac11fa475e56e92f (127.0.145.69:45021)
I20240502 14:01:14.715296 4758 tablet_copy_client.cc:308] T b5172e93df3a4fb5a70188b0507639e6 P 5af426c57bc3434c9fbdcdff873a565c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.69:45021
I20240502 14:01:14.719281 4488 tablet_copy_service.cc:140] P b3988f2f38654eedac11fa475e56e92f: Received BeginTabletCopySession request for tablet b5172e93df3a4fb5a70188b0507639e6 from peer 5af426c57bc3434c9fbdcdff873a565c ({username='slave'} at 127.0.0.1:51194)
I20240502 14:01:14.719367 4488 tablet_copy_service.cc:161] P b3988f2f38654eedac11fa475e56e92f: Beginning new tablet copy session on tablet b5172e93df3a4fb5a70188b0507639e6 from peer 5af426c57bc3434c9fbdcdff873a565c at {username='slave'} at 127.0.0.1:51194: session id = 5af426c57bc3434c9fbdcdff873a565c-b5172e93df3a4fb5a70188b0507639e6
I20240502 14:01:14.719607 4733 raft_consensus.cc:979] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae: : Instructing follower 24fd5d6c84c1483ea8f39bb975e26e45 to start an election
I20240502 14:01:14.719684 4733 raft_consensus.cc:1067] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 LEADER]: Signalling peer 24fd5d6c84c1483ea8f39bb975e26e45 to start an election
I20240502 14:01:14.720005 4078 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "144739b71bf6474d90399816120b3b59"
dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
from {username='slave'} at 127.0.0.1:59794
I20240502 14:01:14.720094 4078 raft_consensus.cc:482] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:14.720149 4078 raft_consensus.cc:3037] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:01:14.720312 4488 tablet_copy_source_session.cc:205] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:14.720909 4078 raft_consensus.cc:504] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:14.721057 4758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5172e93df3a4fb5a70188b0507639e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:14.721243 4078 leader_election.cc:288] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 3 election: Requested vote from peers bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:14.721426 4000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144739b71bf6474d90399816120b3b59" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
I20240502 14:01:14.721521 4000 raft_consensus.cc:3032] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:01:14.721563 4000 raft_consensus.cc:729] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Running, Role: LEADER
I20240502 14:01:14.721676 4000 consensus_queue.cc:260] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:14.721874 4000 raft_consensus.cc:3037] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:01:14.721879 4764 ts_tablet_manager.cc:902] T fb4c3dcde8764e17bb084cb37fef706f P e1de31532912433dbed75c2bb01bc42a: Initiating tablet copy from peer df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:14.722523 4764 tablet_copy_client.cc:308] T fb4c3dcde8764e17bb084cb37fef706f P e1de31532912433dbed75c2bb01bc42a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.67:34309
I20240502 14:01:14.722748 4000 raft_consensus.cc:2454] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 3.
I20240502 14:01:14.722992 4047 leader_election.cc:302] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, bd060dd4339f4fa5a985596bf7dec0ae; no voters:
I20240502 14:01:14.723138 4727 raft_consensus.cc:2781] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:01:14.723202 4727 raft_consensus.cc:686] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 LEADER]: Becoming Leader. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Running, Role: LEADER
I20240502 14:01:14.723259 4154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144739b71bf6474d90399816120b3b59" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "df655d53530449fc85de33de1c229815"
I20240502 14:01:14.723320 4727 consensus_queue.cc:237] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:14.723330 4154 raft_consensus.cc:3037] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:01:14.725076 4154 raft_consensus.cc:2454] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 3.
I20240502 14:01:14.725147 3894 catalog_manager.cc:5484] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 reported cstate change: term changed from 2 to 3, leader changed from bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65) to 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66). New cstate: current_term: 3 leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:14.723199 4758 tablet_copy_client.cc:783] T b5172e93df3a4fb5a70188b0507639e6 P 5af426c57bc3434c9fbdcdff873a565c: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:14.725805 4758 tablet_copy_client.cc:647] T b5172e93df3a4fb5a70188b0507639e6 P 5af426c57bc3434c9fbdcdff873a565c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:14.726949 4762 tablet_copy_client.cc:1002] T b5172e93df3a4fb5a70188b0507639e6 P 5af426c57bc3434c9fbdcdff873a565c: tablet copy: Injecting latency into file download: 2000
I20240502 14:01:14.729099 4167 tablet_copy_service.cc:140] P df655d53530449fc85de33de1c229815: Received BeginTabletCopySession request for tablet fb4c3dcde8764e17bb084cb37fef706f from peer e1de31532912433dbed75c2bb01bc42a ({username='slave'} at 127.0.0.1:33638)
I20240502 14:01:14.729183 4167 tablet_copy_service.cc:161] P df655d53530449fc85de33de1c229815: Beginning new tablet copy session on tablet fb4c3dcde8764e17bb084cb37fef706f from peer e1de31532912433dbed75c2bb01bc42a at {username='slave'} at 127.0.0.1:33638: session id = e1de31532912433dbed75c2bb01bc42a-fb4c3dcde8764e17bb084cb37fef706f
I20240502 14:01:14.729913 4078 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "144739b71bf6474d90399816120b3b59"
dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59772
I20240502 14:01:14.730015 4078 raft_consensus.cc:595] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 LEADER]: Received request to transfer leadership
I20240502 14:01:14.730034 4167 tablet_copy_source_session.cc:205] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:14.730796 4764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb4c3dcde8764e17bb084cb37fef706f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:14.732683 4764 tablet_copy_client.cc:783] T fb4c3dcde8764e17bb084cb37fef706f P e1de31532912433dbed75c2bb01bc42a: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:14.732882 4764 tablet_copy_client.cc:647] T fb4c3dcde8764e17bb084cb37fef706f P e1de31532912433dbed75c2bb01bc42a: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:14.733659 4768 tablet_copy_client.cc:1002] T fb4c3dcde8764e17bb084cb37fef706f P e1de31532912433dbed75c2bb01bc42a: tablet copy: Injecting latency into file download: 2000
I20240502 14:01:14.734602 4078 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "144739b71bf6474d90399816120b3b59"
dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59772
I20240502 14:01:14.734689 4078 raft_consensus.cc:595] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 LEADER]: Received request to transfer leadership
I20240502 14:01:14.735366 4078 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "144739b71bf6474d90399816120b3b59"
dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59772
I20240502 14:01:14.735448 4078 raft_consensus.cc:595] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 LEADER]: Received request to transfer leadership
W20240502 14:01:14.735708 3952 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 144739b71bf6474d90399816120b3b59 already in progress
W20240502 14:01:14.735754 3952 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 144739b71bf6474d90399816120b3b59 already in progress
I20240502 14:01:14.780287 4772 ts_tablet_manager.cc:902] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a: Initiating tablet copy from peer 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251)
I20240502 14:01:14.780799 4772 tablet_copy_client.cc:308] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.66:35251
I20240502 14:01:14.781208 4092 tablet_copy_service.cc:140] P 24fd5d6c84c1483ea8f39bb975e26e45: Received BeginTabletCopySession request for tablet 46b4a50dee854f62aac3a02889525b40 from peer e1de31532912433dbed75c2bb01bc42a ({username='slave'} at 127.0.0.1:59820)
I20240502 14:01:14.781299 4092 tablet_copy_service.cc:161] P 24fd5d6c84c1483ea8f39bb975e26e45: Beginning new tablet copy session on tablet 46b4a50dee854f62aac3a02889525b40 from peer e1de31532912433dbed75c2bb01bc42a at {username='slave'} at 127.0.0.1:59820: session id = e1de31532912433dbed75c2bb01bc42a-46b4a50dee854f62aac3a02889525b40
I20240502 14:01:14.782115 4092 tablet_copy_source_session.cc:205] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:14.782768 4772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46b4a50dee854f62aac3a02889525b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:14.784345 4772 tablet_copy_client.cc:783] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:14.784518 4772 tablet_copy_client.cc:647] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:14.785385 4776 tablet_copy_client.cc:1002] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a: tablet copy: Injecting latency into file download: 2000
I20240502 14:01:14.819224 4777 ts_tablet_manager.cc:902] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 5af426c57bc3434c9fbdcdff873a565c: Initiating tablet copy from peer df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:14.819664 4777 tablet_copy_client.cc:308] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 5af426c57bc3434c9fbdcdff873a565c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.67:34309
I20240502 14:01:14.822994 4167 tablet_copy_service.cc:140] P df655d53530449fc85de33de1c229815: Received BeginTabletCopySession request for tablet 7ac79a5b1f514bdcb36e0db4ff6e3a59 from peer 5af426c57bc3434c9fbdcdff873a565c ({username='slave'} at 127.0.0.1:33652)
I20240502 14:01:14.823081 4167 tablet_copy_service.cc:161] P df655d53530449fc85de33de1c229815: Beginning new tablet copy session on tablet 7ac79a5b1f514bdcb36e0db4ff6e3a59 from peer 5af426c57bc3434c9fbdcdff873a565c at {username='slave'} at 127.0.0.1:33652: session id = 5af426c57bc3434c9fbdcdff873a565c-7ac79a5b1f514bdcb36e0db4ff6e3a59
I20240502 14:01:14.823905 4167 tablet_copy_source_session.cc:205] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:14.824570 4777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ac79a5b1f514bdcb36e0db4ff6e3a59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:14.826418 4777 tablet_copy_client.cc:783] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 5af426c57bc3434c9fbdcdff873a565c: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:14.826567 4777 tablet_copy_client.cc:647] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 5af426c57bc3434c9fbdcdff873a565c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:14.827414 4781 tablet_copy_client.cc:1002] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 5af426c57bc3434c9fbdcdff873a565c: tablet copy: Injecting latency into file download: 2000
I20240502 14:01:14.843097 4727 raft_consensus.cc:979] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45: : Instructing follower bd060dd4339f4fa5a985596bf7dec0ae to start an election
I20240502 14:01:14.843184 4727 raft_consensus.cc:1067] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Signalling peer bd060dd4339f4fa5a985596bf7dec0ae to start an election
I20240502 14:01:14.843510 4000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "46b4a50dee854f62aac3a02889525b40"
dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
from {username='slave'} at 127.0.0.1:57072
I20240502 14:01:14.843622 4000 raft_consensus.cc:482] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:14.843681 4000 raft_consensus.cc:3037] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:14.844347 4000 raft_consensus.cc:504] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } }
I20240502 14:01:14.844645 4000 leader_election.cc:288] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 2 election: Requested vote from peers 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:14.844874 4078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46b4a50dee854f62aac3a02889525b40" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
I20240502 14:01:14.844928 4150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46b4a50dee854f62aac3a02889525b40" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "df655d53530449fc85de33de1c229815"
I20240502 14:01:14.844998 4078 raft_consensus.cc:3032] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:01:14.845033 4078 raft_consensus.cc:729] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Running, Role: LEADER
I20240502 14:01:14.845039 4150 raft_consensus.cc:3037] T 46b4a50dee854f62aac3a02889525b40 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:14.845144 4078 consensus_queue.cc:260] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } }
I20240502 14:01:14.845332 4078 raft_consensus.cc:3037] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:14.845768 4150 raft_consensus.cc:2454] T 46b4a50dee854f62aac3a02889525b40 P df655d53530449fc85de33de1c229815 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 2.
I20240502 14:01:14.845994 4078 raft_consensus.cc:2454] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 2.
I20240502 14:01:14.846031 3972 leader_election.cc:302] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd060dd4339f4fa5a985596bf7dec0ae, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:14.846168 4733 raft_consensus.cc:2781] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:14.846248 4733 raft_consensus.cc:686] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 LEADER]: Becoming Leader. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Running, Role: LEADER
I20240502 14:01:14.846369 4733 consensus_queue.cc:237] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } }
I20240502 14:01:14.848106 3894 catalog_manager.cc:5484] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae reported cstate change: term changed from 1 to 2, leader changed from 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66) to bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65). New cstate: current_term: 2 leader_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:14.909020 4700 raft_consensus.cc:979] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45: : Instructing follower e1de31532912433dbed75c2bb01bc42a to start an election
I20240502 14:01:14.909147 4700 raft_consensus.cc:1067] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Signalling peer e1de31532912433dbed75c2bb01bc42a to start an election
I20240502 14:01:14.909525 4553 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a4d0e9d6d8e7445a9367e0f3a31c8748"
dest_uuid: "e1de31532912433dbed75c2bb01bc42a"
from {username='slave'} at 127.0.0.1:49816
I20240502 14:01:14.909652 4553 raft_consensus.cc:482] T a4d0e9d6d8e7445a9367e0f3a31c8748 P e1de31532912433dbed75c2bb01bc42a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:14.909710 4553 raft_consensus.cc:3037] T a4d0e9d6d8e7445a9367e0f3a31c8748 P e1de31532912433dbed75c2bb01bc42a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:14.910549 4553 raft_consensus.cc:504] T a4d0e9d6d8e7445a9367e0f3a31c8748 P e1de31532912433dbed75c2bb01bc42a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } }
I20240502 14:01:14.911010 4553 leader_election.cc:288] T a4d0e9d6d8e7445a9367e0f3a31c8748 P e1de31532912433dbed75c2bb01bc42a [CANDIDATE]: Term 2 election: Requested vote from peers bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667), 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251)
I20240502 14:01:14.911659 4078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4d0e9d6d8e7445a9367e0f3a31c8748" candidate_uuid: "e1de31532912433dbed75c2bb01bc42a" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
I20240502 14:01:14.911767 4078 raft_consensus.cc:3032] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:01:14.911814 4078 raft_consensus.cc:729] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Running, Role: LEADER
I20240502 14:01:14.911932 4078 consensus_queue.cc:260] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } }
I20240502 14:01:14.912083 4078 raft_consensus.cc:3037] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:14.912801 4078 raft_consensus.cc:2454] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1de31532912433dbed75c2bb01bc42a in term 2.
I20240502 14:01:14.913051 4519 leader_election.cc:302] T a4d0e9d6d8e7445a9367e0f3a31c8748 P e1de31532912433dbed75c2bb01bc42a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, e1de31532912433dbed75c2bb01bc42a; no voters:
I20240502 14:01:14.913357 4783 raft_consensus.cc:2781] T a4d0e9d6d8e7445a9367e0f3a31c8748 P e1de31532912433dbed75c2bb01bc42a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:14.913571 4783 raft_consensus.cc:686] T a4d0e9d6d8e7445a9367e0f3a31c8748 P e1de31532912433dbed75c2bb01bc42a [term 2 LEADER]: Becoming Leader. State: Replica: e1de31532912433dbed75c2bb01bc42a, State: Running, Role: LEADER
I20240502 14:01:14.913700 4783 consensus_queue.cc:237] T a4d0e9d6d8e7445a9367e0f3a31c8748 P e1de31532912433dbed75c2bb01bc42a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } }
I20240502 14:01:14.914810 4000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4d0e9d6d8e7445a9367e0f3a31c8748" candidate_uuid: "e1de31532912433dbed75c2bb01bc42a" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
I20240502 14:01:14.914925 4000 raft_consensus.cc:3037] T a4d0e9d6d8e7445a9367e0f3a31c8748 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:14.915501 3894 catalog_manager.cc:5484] T a4d0e9d6d8e7445a9367e0f3a31c8748 P e1de31532912433dbed75c2bb01bc42a reported cstate change: term changed from 1 to 2, leader changed from 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66) to e1de31532912433dbed75c2bb01bc42a (127.0.145.70). New cstate: current_term: 2 leader_uuid: "e1de31532912433dbed75c2bb01bc42a" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:01:14.915648 4000 raft_consensus.cc:2454] T a4d0e9d6d8e7445a9367e0f3a31c8748 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1de31532912433dbed75c2bb01bc42a in term 2.
I20240502 14:01:15.090291 4150 raft_consensus.cc:1261] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 3 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 14:01:15.090692 4678 consensus_queue.cc:1035] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
W20240502 14:01:15.093044 3969 consensus_peers.cc:485] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae -> Peer b3988f2f38654eedac11fa475e56e92f (127.0.145.69:45021): Couldn't send request to peer b3988f2f38654eedac11fa475e56e92f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:15.093533 4078 raft_consensus.cc:1261] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 14:01:15.093946 4787 consensus_queue.cc:1035] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
W20240502 14:01:15.162896 4045 consensus_peers.cc:485] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 -> Peer b3988f2f38654eedac11fa475e56e92f (127.0.145.69:45021): Couldn't send request to peer b3988f2f38654eedac11fa475e56e92f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 144739b71bf6474d90399816120b3b59. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:15.163286 4000 raft_consensus.cc:1261] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 14:01:15.164122 4700 consensus_queue.cc:1035] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:01:15.165740 4727 raft_consensus.cc:979] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45: : Instructing follower bd060dd4339f4fa5a985596bf7dec0ae to start an election
I20240502 14:01:15.165839 4700 raft_consensus.cc:1067] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 LEADER]: Signalling peer bd060dd4339f4fa5a985596bf7dec0ae to start an election
I20240502 14:01:15.166692 4150 raft_consensus.cc:1261] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 3 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 14:01:15.167032 4700 consensus_queue.cc:1035] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:01:15.168823 4000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "144739b71bf6474d90399816120b3b59"
dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
from {username='slave'} at 127.0.0.1:57072
I20240502 14:01:15.169165 4000 raft_consensus.cc:482] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:15.169237 4000 raft_consensus.cc:3037] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:01:15.170262 4000 raft_consensus.cc:504] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:15.170603 4000 leader_election.cc:288] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 4 election: Requested vote from peers 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:15.170835 4078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144739b71bf6474d90399816120b3b59" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
I20240502 14:01:15.170922 4078 raft_consensus.cc:3032] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:01:15.170967 4078 raft_consensus.cc:729] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Running, Role: LEADER
I20240502 14:01:15.171150 4078 consensus_queue.cc:260] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:15.171331 4078 raft_consensus.cc:3037] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:01:15.170976 4154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144739b71bf6474d90399816120b3b59" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "df655d53530449fc85de33de1c229815"
I20240502 14:01:15.172138 4154 raft_consensus.cc:3037] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:01:15.172873 4154 raft_consensus.cc:2454] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 4.
I20240502 14:01:15.173007 4078 raft_consensus.cc:2454] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 4.
I20240502 14:01:15.173264 3972 leader_election.cc:302] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, bd060dd4339f4fa5a985596bf7dec0ae; no voters:
I20240502 14:01:15.173498 4787 raft_consensus.cc:2781] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:01:15.173585 4787 raft_consensus.cc:686] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 4 LEADER]: Becoming Leader. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Running, Role: LEADER
I20240502 14:01:15.173710 4787 consensus_queue.cc:237] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:15.175033 3894 catalog_manager.cc:5484] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae reported cstate change: term changed from 3 to 4, leader changed from 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66) to bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65). New cstate: current_term: 4 leader_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:15.229280 4150 raft_consensus.cc:1261] T 46b4a50dee854f62aac3a02889525b40 P df655d53530449fc85de33de1c229815 [term 2 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 14:01:15.229652 4678 consensus_queue.cc:1035] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:01:15.232079 4078 raft_consensus.cc:1261] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 14:01:15.232641 4787 consensus_queue.cc:1035] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
W20240502 14:01:15.235649 3969 consensus_peers.cc:485] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae -> Peer e1de31532912433dbed75c2bb01bc42a (127.0.145.70:41637): Couldn't send request to peer e1de31532912433dbed75c2bb01bc42a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:15.255818 4154 raft_consensus.cc:1261] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [term 3 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 14:01:15.256146 4700 consensus_queue.cc:1035] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:01:15.258587 4478 raft_consensus.cc:1261] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f [term 3 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 14:01:15.258929 4793 consensus_queue.cc:1035] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:01:15.344637 4005 raft_consensus.cc:1261] T a4d0e9d6d8e7445a9367e0f3a31c8748 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Refusing update from remote peer e1de31532912433dbed75c2bb01bc42a: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 14:01:15.345021 4783 consensus_queue.cc:1035] T a4d0e9d6d8e7445a9367e0f3a31c8748 P e1de31532912433dbed75c2bb01bc42a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:01:15.347222 4078 raft_consensus.cc:1261] T a4d0e9d6d8e7445a9367e0f3a31c8748 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Refusing update from remote peer e1de31532912433dbed75c2bb01bc42a: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 14:01:15.347601 4784 consensus_queue.cc:1035] T a4d0e9d6d8e7445a9367e0f3a31c8748 P e1de31532912433dbed75c2bb01bc42a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
W20240502 14:01:15.584080 3969 consensus_peers.cc:485] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae -> Peer b3988f2f38654eedac11fa475e56e92f (127.0.145.69:45021): Couldn't send request to peer b3988f2f38654eedac11fa475e56e92f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 144739b71bf6474d90399816120b3b59. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:15.693337 4078 raft_consensus.cc:1261] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 4 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240502 14:01:15.693691 4733 consensus_queue.cc:1035] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:01:15.695976 4154 raft_consensus.cc:1261] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 4 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240502 14:01:15.696310 4733 consensus_queue.cc:1035] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:01:16.153231 4807 ts_tablet_manager.cc:902] T 144739b71bf6474d90399816120b3b59 P b3988f2f38654eedac11fa475e56e92f: Initiating tablet copy from peer bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667)
I20240502 14:01:16.153748 4807 tablet_copy_client.cc:308] T 144739b71bf6474d90399816120b3b59 P b3988f2f38654eedac11fa475e56e92f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.65:46667
I20240502 14:01:16.154145 4016 tablet_copy_service.cc:140] P bd060dd4339f4fa5a985596bf7dec0ae: Received BeginTabletCopySession request for tablet 144739b71bf6474d90399816120b3b59 from peer b3988f2f38654eedac11fa475e56e92f ({username='slave'} at 127.0.0.1:57094)
I20240502 14:01:16.154237 4016 tablet_copy_service.cc:161] P bd060dd4339f4fa5a985596bf7dec0ae: Beginning new tablet copy session on tablet 144739b71bf6474d90399816120b3b59 from peer b3988f2f38654eedac11fa475e56e92f at {username='slave'} at 127.0.0.1:57094: session id = b3988f2f38654eedac11fa475e56e92f-144739b71bf6474d90399816120b3b59
I20240502 14:01:16.155192 4016 tablet_copy_source_session.cc:205] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:16.155912 4807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 144739b71bf6474d90399816120b3b59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:16.158047 4807 tablet_copy_client.cc:783] T 144739b71bf6474d90399816120b3b59 P b3988f2f38654eedac11fa475e56e92f: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:16.158197 4807 tablet_copy_client.cc:647] T 144739b71bf6474d90399816120b3b59 P b3988f2f38654eedac11fa475e56e92f: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:16.159072 4811 tablet_copy_client.cc:1002] T 144739b71bf6474d90399816120b3b59 P b3988f2f38654eedac11fa475e56e92f: tablet copy: Injecting latency into file download: 2000
I20240502 14:01:16.678962 3957 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:01:16.679453 4752 tablet_copy_client.cc:523] T c965f0e11bc1433f8ccdc20c98b4b4b9 P b3988f2f38654eedac11fa475e56e92f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:16.680809 4000 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "144739b71bf6474d90399816120b3b59"
dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
mode: GRACEFUL
new_leader_uuid: "df655d53530449fc85de33de1c229815"
from {username='slave'} at 127.0.0.1:57080
I20240502 14:01:16.680943 4000 raft_consensus.cc:595] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 4 LEADER]: Received request to transfer leadership to TS df655d53530449fc85de33de1c229815
I20240502 14:01:16.681557 4078 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3d0d4aeaec264d04bae2c34e6edc6ec7"
dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
mode: GRACEFUL
new_leader_uuid: "5af426c57bc3434c9fbdcdff873a565c"
from {username='slave'} at 127.0.0.1:59812
I20240502 14:01:16.681659 4078 raft_consensus.cc:595] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Received request to transfer leadership to TS 5af426c57bc3434c9fbdcdff873a565c
I20240502 14:01:16.682262 4000 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "975becfb8df0430fa8c748a58dcbcfb1"
dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
mode: GRACEFUL
new_leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
from {username='slave'} at 127.0.0.1:57080
I20240502 14:01:16.682395 4000 raft_consensus.cc:595] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 LEADER]: Received request to transfer leadership to TS 24fd5d6c84c1483ea8f39bb975e26e45
I20240502 14:01:16.682790 4752 tablet_bootstrap.cc:492] T c965f0e11bc1433f8ccdc20c98b4b4b9 P b3988f2f38654eedac11fa475e56e92f: Bootstrap starting.
I20240502 14:01:16.683040 4000 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c965f0e11bc1433f8ccdc20c98b4b4b9"
dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
mode: GRACEFUL
new_leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
from {username='slave'} at 127.0.0.1:57080
I20240502 14:01:16.683148 4000 raft_consensus.cc:595] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 LEADER]: Received request to transfer leadership to TS 24fd5d6c84c1483ea8f39bb975e26e45
I20240502 14:01:16.683419 3957 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240502 14:01:16.683549 3957 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:01:16.691421 4812 raft_consensus.cc:979] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae: : Instructing follower df655d53530449fc85de33de1c229815 to start an election
I20240502 14:01:16.691502 4812 raft_consensus.cc:1067] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 4 LEADER]: Signalling peer df655d53530449fc85de33de1c229815 to start an election
I20240502 14:01:16.691818 4150 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "144739b71bf6474d90399816120b3b59"
dest_uuid: "df655d53530449fc85de33de1c229815"
from {username='slave'} at 127.0.0.1:33594
I20240502 14:01:16.691931 4150 raft_consensus.cc:482] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:16.691992 4150 raft_consensus.cc:3037] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:01:16.692988 4150 raft_consensus.cc:504] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:16.693358 4150 leader_election.cc:288] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [CANDIDATE]: Term 5 election: Requested vote from peers bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667), 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251)
I20240502 14:01:16.693567 4000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144739b71bf6474d90399816120b3b59" candidate_uuid: "df655d53530449fc85de33de1c229815" candidate_term: 5 candidate_status { last_received { term: 4 index: 31 } } ignore_live_leader: true dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
I20240502 14:01:16.693601 4078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144739b71bf6474d90399816120b3b59" candidate_uuid: "df655d53530449fc85de33de1c229815" candidate_term: 5 candidate_status { last_received { term: 4 index: 31 } } ignore_live_leader: true dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
I20240502 14:01:16.693683 4000 raft_consensus.cc:3032] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 4 LEADER]: Stepping down as leader of term 4
I20240502 14:01:16.693683 4078 raft_consensus.cc:3037] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:01:16.693737 4000 raft_consensus.cc:729] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 4 LEADER]: Becoming Follower/Learner. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Running, Role: LEADER
I20240502 14:01:16.693871 4000 consensus_queue.cc:260] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 31, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:16.694036 4000 raft_consensus.cc:3037] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:01:16.694604 4078 raft_consensus.cc:2454] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df655d53530449fc85de33de1c229815 in term 5.
I20240502 14:01:16.694844 4000 raft_consensus.cc:2454] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df655d53530449fc85de33de1c229815 in term 5.
I20240502 14:01:16.694890 4123 leader_election.cc:302] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:16.695037 4688 raft_consensus.cc:2781] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 14:01:16.695186 4688 raft_consensus.cc:686] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 5 LEADER]: Becoming Leader. State: Replica: df655d53530449fc85de33de1c229815, State: Running, Role: LEADER
I20240502 14:01:16.695331 4688 consensus_queue.cc:237] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 31, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:16.696974 3903 catalog_manager.cc:5484] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 reported cstate change: term changed from 4 to 5, leader changed from bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65) to df655d53530449fc85de33de1c229815 (127.0.145.67). New cstate: current_term: 5 leader_uuid: "df655d53530449fc85de33de1c229815" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:16.706097 4752 tablet_bootstrap.cc:492] T c965f0e11bc1433f8ccdc20c98b4b4b9 P b3988f2f38654eedac11fa475e56e92f: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=118 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:16.706349 4752 tablet_bootstrap.cc:492] T c965f0e11bc1433f8ccdc20c98b4b4b9 P b3988f2f38654eedac11fa475e56e92f: Bootstrap complete.
I20240502 14:01:16.706521 4752 ts_tablet_manager.cc:1367] T c965f0e11bc1433f8ccdc20c98b4b4b9 P b3988f2f38654eedac11fa475e56e92f: Time spent bootstrapping tablet: real 0.024s user 0.018s sys 0.004s
I20240502 14:01:16.706956 4752 raft_consensus.cc:348] T c965f0e11bc1433f8ccdc20c98b4b4b9 P b3988f2f38654eedac11fa475e56e92f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:16.707057 4752 raft_consensus.cc:729] T c965f0e11bc1433f8ccdc20c98b4b4b9 P b3988f2f38654eedac11fa475e56e92f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b3988f2f38654eedac11fa475e56e92f, State: Initialized, Role: LEARNER
I20240502 14:01:16.707141 4752 consensus_queue.cc:260] T c965f0e11bc1433f8ccdc20c98b4b4b9 P b3988f2f38654eedac11fa475e56e92f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:16.708542 4752 ts_tablet_manager.cc:1398] T c965f0e11bc1433f8ccdc20c98b4b4b9 P b3988f2f38654eedac11fa475e56e92f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:01:16.709182 4167 tablet_copy_service.cc:342] P df655d53530449fc85de33de1c229815: Request end of tablet copy session b3988f2f38654eedac11fa475e56e92f-c965f0e11bc1433f8ccdc20c98b4b4b9 received from {username='slave'} at 127.0.0.1:33622
I20240502 14:01:16.709254 4167 tablet_copy_service.cc:433] P df655d53530449fc85de33de1c229815: ending tablet copy session b3988f2f38654eedac11fa475e56e92f-c965f0e11bc1433f8ccdc20c98b4b4b9 on tablet c965f0e11bc1433f8ccdc20c98b4b4b9 with peer b3988f2f38654eedac11fa475e56e92f
I20240502 14:01:16.727203 4758 tablet_copy_client.cc:523] T b5172e93df3a4fb5a70188b0507639e6 P 5af426c57bc3434c9fbdcdff873a565c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:16.728549 4758 tablet_bootstrap.cc:492] T b5172e93df3a4fb5a70188b0507639e6 P 5af426c57bc3434c9fbdcdff873a565c: Bootstrap starting.
I20240502 14:01:16.733927 4764 tablet_copy_client.cc:523] T fb4c3dcde8764e17bb084cb37fef706f P e1de31532912433dbed75c2bb01bc42a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:16.734928 4764 tablet_bootstrap.cc:492] T fb4c3dcde8764e17bb084cb37fef706f P e1de31532912433dbed75c2bb01bc42a: Bootstrap starting.
I20240502 14:01:16.745405 4804 raft_consensus.cc:979] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45: : Instructing follower 5af426c57bc3434c9fbdcdff873a565c to start an election
I20240502 14:01:16.745486 4804 raft_consensus.cc:1067] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Signalling peer 5af426c57bc3434c9fbdcdff873a565c to start an election
I20240502 14:01:16.745824 4401 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3d0d4aeaec264d04bae2c34e6edc6ec7"
dest_uuid: "5af426c57bc3434c9fbdcdff873a565c"
from {username='slave'} at 127.0.0.1:55470
I20240502 14:01:16.745949 4401 raft_consensus.cc:482] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 5af426c57bc3434c9fbdcdff873a565c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:16.746008 4401 raft_consensus.cc:3037] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 5af426c57bc3434c9fbdcdff873a565c [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:16.746894 4401 raft_consensus.cc:504] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 5af426c57bc3434c9fbdcdff873a565c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } }
I20240502 14:01:16.747335 4401 leader_election.cc:288] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 5af426c57bc3434c9fbdcdff873a565c [CANDIDATE]: Term 2 election: Requested vote from peers 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:16.747822 4078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d0d4aeaec264d04bae2c34e6edc6ec7" candidate_uuid: "5af426c57bc3434c9fbdcdff873a565c" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
I20240502 14:01:16.747920 4078 raft_consensus.cc:3032] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:01:16.747956 4078 raft_consensus.cc:729] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Running, Role: LEADER
I20240502 14:01:16.748090 4078 consensus_queue.cc:260] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } }
I20240502 14:01:16.748225 4078 raft_consensus.cc:3037] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:16.749104 4078 raft_consensus.cc:2454] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5af426c57bc3434c9fbdcdff873a565c in term 2.
I20240502 14:01:16.749325 4150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d0d4aeaec264d04bae2c34e6edc6ec7" candidate_uuid: "5af426c57bc3434c9fbdcdff873a565c" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "df655d53530449fc85de33de1c229815"
I20240502 14:01:16.749426 4150 raft_consensus.cc:3037] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:16.750270 4150 raft_consensus.cc:2454] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P df655d53530449fc85de33de1c229815 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5af426c57bc3434c9fbdcdff873a565c in term 2.
I20240502 14:01:16.750558 4366 leader_election.cc:302] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 5af426c57bc3434c9fbdcdff873a565c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5af426c57bc3434c9fbdcdff873a565c, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:16.750806 4823 raft_consensus.cc:2781] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 5af426c57bc3434c9fbdcdff873a565c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:16.751016 4758 tablet_bootstrap.cc:492] T b5172e93df3a4fb5a70188b0507639e6 P 5af426c57bc3434c9fbdcdff873a565c: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=83 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:16.751030 4823 raft_consensus.cc:686] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 5af426c57bc3434c9fbdcdff873a565c [term 2 LEADER]: Becoming Leader. State: Replica: 5af426c57bc3434c9fbdcdff873a565c, State: Running, Role: LEADER
I20240502 14:01:16.751291 4823 consensus_queue.cc:237] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 5af426c57bc3434c9fbdcdff873a565c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } }
I20240502 14:01:16.751366 4758 tablet_bootstrap.cc:492] T b5172e93df3a4fb5a70188b0507639e6 P 5af426c57bc3434c9fbdcdff873a565c: Bootstrap complete.
I20240502 14:01:16.751639 4758 ts_tablet_manager.cc:1367] T b5172e93df3a4fb5a70188b0507639e6 P 5af426c57bc3434c9fbdcdff873a565c: Time spent bootstrapping tablet: real 0.023s user 0.020s sys 0.004s
I20240502 14:01:16.752455 4758 raft_consensus.cc:348] T b5172e93df3a4fb5a70188b0507639e6 P 5af426c57bc3434c9fbdcdff873a565c [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } }
I20240502 14:01:16.752621 4758 raft_consensus.cc:729] T b5172e93df3a4fb5a70188b0507639e6 P 5af426c57bc3434c9fbdcdff873a565c [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 5af426c57bc3434c9fbdcdff873a565c, State: Initialized, Role: LEARNER
I20240502 14:01:16.752792 4758 consensus_queue.cc:260] T b5172e93df3a4fb5a70188b0507639e6 P 5af426c57bc3434c9fbdcdff873a565c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 4.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } }
I20240502 14:01:16.752889 3894 catalog_manager.cc:5484] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 5af426c57bc3434c9fbdcdff873a565c reported cstate change: term changed from 1 to 2, leader changed from 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66) to 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68). New cstate: current_term: 2 leader_uuid: "5af426c57bc3434c9fbdcdff873a565c" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:01:16.753186 4758 ts_tablet_manager.cc:1398] T b5172e93df3a4fb5a70188b0507639e6 P 5af426c57bc3434c9fbdcdff873a565c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:16.753636 4488 tablet_copy_service.cc:342] P b3988f2f38654eedac11fa475e56e92f: Request end of tablet copy session 5af426c57bc3434c9fbdcdff873a565c-b5172e93df3a4fb5a70188b0507639e6 received from {username='slave'} at 127.0.0.1:51194
I20240502 14:01:16.753698 4488 tablet_copy_service.cc:433] P b3988f2f38654eedac11fa475e56e92f: ending tablet copy session 5af426c57bc3434c9fbdcdff873a565c-b5172e93df3a4fb5a70188b0507639e6 on tablet b5172e93df3a4fb5a70188b0507639e6 with peer 5af426c57bc3434c9fbdcdff873a565c
I20240502 14:01:16.757896 4764 tablet_bootstrap.cc:492] T fb4c3dcde8764e17bb084cb37fef706f P e1de31532912433dbed75c2bb01bc42a: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=114 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:16.758108 4764 tablet_bootstrap.cc:492] T fb4c3dcde8764e17bb084cb37fef706f P e1de31532912433dbed75c2bb01bc42a: Bootstrap complete.
I20240502 14:01:16.758256 4764 ts_tablet_manager.cc:1367] T fb4c3dcde8764e17bb084cb37fef706f P e1de31532912433dbed75c2bb01bc42a: Time spent bootstrapping tablet: real 0.023s user 0.021s sys 0.000s
I20240502 14:01:16.758600 4764 raft_consensus.cc:348] T fb4c3dcde8764e17bb084cb37fef706f P e1de31532912433dbed75c2bb01bc42a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } }
I20240502 14:01:16.758697 4764 raft_consensus.cc:729] T fb4c3dcde8764e17bb084cb37fef706f P e1de31532912433dbed75c2bb01bc42a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e1de31532912433dbed75c2bb01bc42a, State: Initialized, Role: LEARNER
I20240502 14:01:16.758790 4764 consensus_queue.cc:260] T fb4c3dcde8764e17bb084cb37fef706f P e1de31532912433dbed75c2bb01bc42a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } }
I20240502 14:01:16.759016 4764 ts_tablet_manager.cc:1398] T fb4c3dcde8764e17bb084cb37fef706f P e1de31532912433dbed75c2bb01bc42a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:16.759425 4167 tablet_copy_service.cc:342] P df655d53530449fc85de33de1c229815: Request end of tablet copy session e1de31532912433dbed75c2bb01bc42a-fb4c3dcde8764e17bb084cb37fef706f received from {username='slave'} at 127.0.0.1:33638
I20240502 14:01:16.759491 4167 tablet_copy_service.cc:433] P df655d53530449fc85de33de1c229815: ending tablet copy session e1de31532912433dbed75c2bb01bc42a-fb4c3dcde8764e17bb084cb37fef706f on tablet fb4c3dcde8764e17bb084cb37fef706f with peer e1de31532912433dbed75c2bb01bc42a
I20240502 14:01:16.785614 4772 tablet_copy_client.cc:523] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:16.786904 4772 tablet_bootstrap.cc:492] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a: Bootstrap starting.
I20240502 14:01:16.808200 4772 tablet_bootstrap.cc:492] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=115 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:16.808413 4772 tablet_bootstrap.cc:492] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a: Bootstrap complete.
I20240502 14:01:16.808574 4772 ts_tablet_manager.cc:1367] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a: Time spent bootstrapping tablet: real 0.022s user 0.015s sys 0.004s
I20240502 14:01:16.808982 4772 raft_consensus.cc:348] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } }
I20240502 14:01:16.809082 4772 raft_consensus.cc:729] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e1de31532912433dbed75c2bb01bc42a, State: Initialized, Role: LEARNER
I20240502 14:01:16.809171 4772 consensus_queue.cc:260] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } }
I20240502 14:01:16.809449 4772 ts_tablet_manager.cc:1398] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 14:01:16.809922 4092 tablet_copy_service.cc:342] P 24fd5d6c84c1483ea8f39bb975e26e45: Request end of tablet copy session e1de31532912433dbed75c2bb01bc42a-46b4a50dee854f62aac3a02889525b40 received from {username='slave'} at 127.0.0.1:59820
I20240502 14:01:16.809985 4092 tablet_copy_service.cc:433] P 24fd5d6c84c1483ea8f39bb975e26e45: ending tablet copy session e1de31532912433dbed75c2bb01bc42a-46b4a50dee854f62aac3a02889525b40 on tablet 46b4a50dee854f62aac3a02889525b40 with peer e1de31532912433dbed75c2bb01bc42a
I20240502 14:01:16.822377 4401 raft_consensus.cc:1203] T b5172e93df3a4fb5a70188b0507639e6 P 5af426c57bc3434c9fbdcdff873a565c [term 4 LEARNER]: Deduplicated request from leader. Original: 4.31->[4.32-4.32] Dedup: 4.32->[]
I20240502 14:01:16.827764 4777 tablet_copy_client.cc:523] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 5af426c57bc3434c9fbdcdff873a565c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:16.829035 4777 tablet_bootstrap.cc:492] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 5af426c57bc3434c9fbdcdff873a565c: Bootstrap starting.
I20240502 14:01:16.849367 4777 tablet_bootstrap.cc:492] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 5af426c57bc3434c9fbdcdff873a565c: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=106 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:16.849628 4777 tablet_bootstrap.cc:492] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 5af426c57bc3434c9fbdcdff873a565c: Bootstrap complete.
I20240502 14:01:16.849794 4777 ts_tablet_manager.cc:1367] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 5af426c57bc3434c9fbdcdff873a565c: Time spent bootstrapping tablet: real 0.021s user 0.020s sys 0.000s
I20240502 14:01:16.850149 4777 raft_consensus.cc:348] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 5af426c57bc3434c9fbdcdff873a565c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } }
I20240502 14:01:16.850258 4777 raft_consensus.cc:729] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 5af426c57bc3434c9fbdcdff873a565c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5af426c57bc3434c9fbdcdff873a565c, State: Initialized, Role: LEARNER
I20240502 14:01:16.850338 4777 consensus_queue.cc:260] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 5af426c57bc3434c9fbdcdff873a565c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } }
I20240502 14:01:16.850569 4777 ts_tablet_manager.cc:1398] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 5af426c57bc3434c9fbdcdff873a565c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:16.851030 4167 tablet_copy_service.cc:342] P df655d53530449fc85de33de1c229815: Request end of tablet copy session 5af426c57bc3434c9fbdcdff873a565c-7ac79a5b1f514bdcb36e0db4ff6e3a59 received from {username='slave'} at 127.0.0.1:33652
I20240502 14:01:16.851100 4167 tablet_copy_service.cc:433] P df655d53530449fc85de33de1c229815: ending tablet copy session 5af426c57bc3434c9fbdcdff873a565c-7ac79a5b1f514bdcb36e0db4ff6e3a59 on tablet 7ac79a5b1f514bdcb36e0db4ff6e3a59 with peer 5af426c57bc3434c9fbdcdff873a565c
I20240502 14:01:16.872103 4478 raft_consensus.cc:3037] T c965f0e11bc1433f8ccdc20c98b4b4b9 P b3988f2f38654eedac11fa475e56e92f [term 2 LEARNER]: Advancing to term 3
I20240502 14:01:17.082015 4812 raft_consensus.cc:979] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae: : Instructing follower 24fd5d6c84c1483ea8f39bb975e26e45 to start an election
I20240502 14:01:17.082145 4812 raft_consensus.cc:1067] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 LEADER]: Signalling peer 24fd5d6c84c1483ea8f39bb975e26e45 to start an election
I20240502 14:01:17.082501 4078 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "975becfb8df0430fa8c748a58dcbcfb1"
dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
from {username='slave'} at 127.0.0.1:59794
I20240502 14:01:17.082615 4078 raft_consensus.cc:482] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:17.082671 4078 raft_consensus.cc:3037] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:17.083518 4078 raft_consensus.cc:504] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:17.083878 4078 leader_election.cc:288] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 2 election: Requested vote from peers bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:17.084097 4150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975becfb8df0430fa8c748a58dcbcfb1" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "df655d53530449fc85de33de1c229815"
I20240502 14:01:17.084110 4005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975becfb8df0430fa8c748a58dcbcfb1" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
I20240502 14:01:17.084224 4150 raft_consensus.cc:3037] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:17.084237 4005 raft_consensus.cc:3032] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:01:17.084285 4005 raft_consensus.cc:729] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Running, Role: LEADER
I20240502 14:01:17.084422 4005 consensus_queue.cc:260] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:17.084579 4005 raft_consensus.cc:3037] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:17.085175 4150 raft_consensus.cc:2454] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 2.
I20240502 14:01:17.085475 4048 leader_election.cc:302] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:17.085512 4005 raft_consensus.cc:2454] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 2.
I20240502 14:01:17.085642 4822 raft_consensus.cc:2781] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:17.085716 4822 raft_consensus.cc:686] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 LEADER]: Becoming Leader. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Running, Role: LEADER
I20240502 14:01:17.085824 4822 consensus_queue.cc:237] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:17.087190 3894 catalog_manager.cc:5484] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 reported cstate change: term changed from 1 to 2, leader changed from bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65) to 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66). New cstate: current_term: 2 leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:17.147207 4150 raft_consensus.cc:1261] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P df655d53530449fc85de33de1c229815 [term 2 FOLLOWER]: Refusing update from remote peer 5af426c57bc3434c9fbdcdff873a565c: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 14:01:17.147583 4824 consensus_queue.cc:1035] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 5af426c57bc3434c9fbdcdff873a565c [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:01:17.150401 4078 raft_consensus.cc:1261] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Refusing update from remote peer 5af426c57bc3434c9fbdcdff873a565c: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 14:01:17.150409 4082 raft_consensus.cc:1261] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 5 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20240502 14:01:17.150805 4688 consensus_queue.cc:1035] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 14:01:17.150861 4823 consensus_queue.cc:1035] T 3d0d4aeaec264d04bae2c34e6edc6ec7 P 5af426c57bc3434c9fbdcdff873a565c [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
W20240502 14:01:17.153549 4120 consensus_peers.cc:485] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 -> Peer b3988f2f38654eedac11fa475e56e92f (127.0.145.69:45021): Couldn't send request to peer b3988f2f38654eedac11fa475e56e92f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:17.153784 4005 raft_consensus.cc:1261] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 5 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20240502 14:01:17.154135 4688 consensus_queue.cc:1035] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 14:01:17.173277 4553 raft_consensus.cc:1203] T fb4c3dcde8764e17bb084cb37fef706f P e1de31532912433dbed75c2bb01bc42a [term 2 LEARNER]: Deduplicated request from leader. Original: 2.28->[2.29-2.29] Dedup: 2.29->[]
I20240502 14:01:17.181416 4812 raft_consensus.cc:979] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae: : Instructing follower 24fd5d6c84c1483ea8f39bb975e26e45 to start an election
I20240502 14:01:17.181510 4812 raft_consensus.cc:1067] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 LEADER]: Signalling peer 24fd5d6c84c1483ea8f39bb975e26e45 to start an election
I20240502 14:01:17.181876 4082 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c965f0e11bc1433f8ccdc20c98b4b4b9"
dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
from {username='slave'} at 127.0.0.1:59794
I20240502 14:01:17.181995 4082 raft_consensus.cc:482] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:17.182058 4082 raft_consensus.cc:3037] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:01:17.182974 4082 raft_consensus.cc:504] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:17.183287 4082 leader_election.cc:288] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 4 election: Requested vote from peers bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:17.183511 4005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c965f0e11bc1433f8ccdc20c98b4b4b9" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
I20240502 14:01:17.183641 4005 raft_consensus.cc:3032] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:01:17.183689 4005 raft_consensus.cc:729] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Running, Role: LEADER
I20240502 14:01:17.183808 4005 consensus_queue.cc:260] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:17.183960 4005 raft_consensus.cc:3037] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:01:17.184594 4005 raft_consensus.cc:2454] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 4.
I20240502 14:01:17.184804 4150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c965f0e11bc1433f8ccdc20c98b4b4b9" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "df655d53530449fc85de33de1c229815"
I20240502 14:01:17.184890 4150 raft_consensus.cc:3037] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:01:17.185089 4047 leader_election.cc:302] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, bd060dd4339f4fa5a985596bf7dec0ae; no voters:
I20240502 14:01:17.185245 4822 raft_consensus.cc:2781] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:01:17.185328 4822 raft_consensus.cc:686] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 4 LEADER]: Becoming Leader. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Running, Role: LEADER
I20240502 14:01:17.185449 4822 consensus_queue.cc:237] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:17.185711 4150 raft_consensus.cc:2454] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 4.
I20240502 14:01:17.186820 3894 catalog_manager.cc:5484] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 reported cstate change: term changed from 3 to 4, leader changed from bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65) to 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66). New cstate: current_term: 4 leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:17.189409 4082 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c965f0e11bc1433f8ccdc20c98b4b4b9"
dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59772
I20240502 14:01:17.189494 4082 raft_consensus.cc:595] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 4 LEADER]: Received request to transfer leadership
I20240502 14:01:17.193163 4082 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c965f0e11bc1433f8ccdc20c98b4b4b9"
dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59772
I20240502 14:01:17.193248 4082 raft_consensus.cc:595] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 4 LEADER]: Received request to transfer leadership
W20240502 14:01:17.193413 3890 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for c965f0e11bc1433f8ccdc20c98b4b4b9 already in progress [suppressed 1 similar messages]
I20240502 14:01:17.193885 4082 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c965f0e11bc1433f8ccdc20c98b4b4b9"
dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59772
I20240502 14:01:17.193969 4082 raft_consensus.cc:595] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 4 LEADER]: Received request to transfer leadership
W20240502 14:01:17.194196 3952 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for c965f0e11bc1433f8ccdc20c98b4b4b9 already in progress
W20240502 14:01:17.194247 3952 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for c965f0e11bc1433f8ccdc20c98b4b4b9 already in progress
I20240502 14:01:17.195220 4401 raft_consensus.cc:1203] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 5af426c57bc3434c9fbdcdff873a565c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20240502 14:01:17.242763 4553 raft_consensus.cc:3037] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a [term 1 LEARNER]: Advancing to term 2
I20240502 14:01:17.317183 4845 raft_consensus.cc:1050] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f: attempting to promote NON_VOTER 5af426c57bc3434c9fbdcdff873a565c to VOTER
I20240502 14:01:17.317631 4845 consensus_queue.cc:237] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 30, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } }
I20240502 14:01:17.318790 4401 raft_consensus.cc:1261] T b5172e93df3a4fb5a70188b0507639e6 P 5af426c57bc3434c9fbdcdff873a565c [term 4 LEARNER]: Refusing update from remote peer b3988f2f38654eedac11fa475e56e92f: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240502 14:01:17.318809 4150 raft_consensus.cc:1261] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815 [term 4 FOLLOWER]: Refusing update from remote peer b3988f2f38654eedac11fa475e56e92f: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240502 14:01:17.319177 4000 raft_consensus.cc:1261] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 4 FOLLOWER]: Refusing update from remote peer b3988f2f38654eedac11fa475e56e92f: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240502 14:01:17.319337 4803 consensus_queue.cc:1035] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 14:01:17.319499 4847 consensus_queue.cc:1035] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 14:01:17.319677 4844 consensus_queue.cc:1035] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 14:01:17.321449 4803 raft_consensus.cc:2932] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [term 4 LEADER]: Committing config change with OpId 4.33: config changed from index 32 to 33, 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } }
I20240502 14:01:17.322247 4154 raft_consensus.cc:2932] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 32 to 33, 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } }
I20240502 14:01:17.322973 4401 raft_consensus.cc:2932] T b5172e93df3a4fb5a70188b0507639e6 P 5af426c57bc3434c9fbdcdff873a565c [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 32 to 33, 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } }
I20240502 14:01:17.323982 4000 raft_consensus.cc:2932] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 32 to 33, 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } }
I20240502 14:01:17.325122 3900 catalog_manager.cc:5484] T b5172e93df3a4fb5a70188b0507639e6 P 5af426c57bc3434c9fbdcdff873a565c reported cstate change: config changed from index 32 to 33, 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "b3988f2f38654eedac11fa475e56e92f" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } }
I20240502 14:01:17.328382 4478 consensus_queue.cc:237] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } }
I20240502 14:01:17.329396 4000 raft_consensus.cc:1261] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 4 FOLLOWER]: Refusing update from remote peer b3988f2f38654eedac11fa475e56e92f: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20240502 14:01:17.329782 4401 raft_consensus.cc:1261] T b5172e93df3a4fb5a70188b0507639e6 P 5af426c57bc3434c9fbdcdff873a565c [term 4 FOLLOWER]: Refusing update from remote peer b3988f2f38654eedac11fa475e56e92f: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20240502 14:01:17.329818 4847 consensus_queue.cc:1035] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 14:01:17.330125 4844 consensus_queue.cc:1035] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 14:01:17.331585 4845 raft_consensus.cc:2932] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [term 4 LEADER]: Committing config change with OpId 4.34: config changed from index 33 to 34, VOTER df655d53530449fc85de33de1c229815 (127.0.145.67) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } }
I20240502 14:01:17.331883 4401 raft_consensus.cc:2932] T b5172e93df3a4fb5a70188b0507639e6 P 5af426c57bc3434c9fbdcdff873a565c [term 4 FOLLOWER]: Committing config change with OpId 4.34: config changed from index 33 to 34, VOTER df655d53530449fc85de33de1c229815 (127.0.145.67) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } }
I20240502 14:01:17.332844 3946 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b5172e93df3a4fb5a70188b0507639e6 with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:01:17.333127 4005 raft_consensus.cc:2932] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 4 FOLLOWER]: Committing config change with OpId 4.34: config changed from index 33 to 34, VOTER df655d53530449fc85de33de1c229815 (127.0.145.67) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } }
I20240502 14:01:17.333604 3903 catalog_manager.cc:5484] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f reported cstate change: config changed from index 33 to 34, VOTER df655d53530449fc85de33de1c229815 (127.0.145.67) evicted. New cstate: current_term: 4 leader_uuid: "b3988f2f38654eedac11fa475e56e92f" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:01:17.335749 4142 tablet_service.cc:1514] Processing DeleteTablet for tablet b5172e93df3a4fb5a70188b0507639e6 with delete_type TABLET_DATA_TOMBSTONED (TS df655d53530449fc85de33de1c229815 not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:33568
I20240502 14:01:17.336020 4851 tablet_replica.cc:330] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815: stopping tablet replica
I20240502 14:01:17.336169 4851 raft_consensus.cc:2229] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:17.336316 4851 raft_consensus.cc:2258] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:17.336825 4851 ts_tablet_manager.cc:1875] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:17.338624 4851 ts_tablet_manager.cc:1888] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.33
I20240502 14:01:17.338688 4851 log.cc:1196] T b5172e93df3a4fb5a70188b0507639e6 P df655d53530449fc85de33de1c229815: Deleting WAL directory at /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-2-root/wals/b5172e93df3a4fb5a70188b0507639e6
I20240502 14:01:17.339133 3890 catalog_manager.cc:4830] TS df655d53530449fc85de33de1c229815 (127.0.145.67:34309): tablet b5172e93df3a4fb5a70188b0507639e6 (table test-workload [id=66735bdbe8fc4765af013422ce730ad0]) successfully deleted
I20240502 14:01:17.600991 4154 raft_consensus.cc:1261] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [term 2 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 14:01:17.601444 4804 consensus_queue.cc:1035] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:01:17.604198 4005 raft_consensus.cc:1261] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 14:01:17.604583 4804 consensus_queue.cc:1035] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:01:17.611546 4154 raft_consensus.cc:1261] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 4 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240502 14:01:17.611872 4804 consensus_queue.cc:1035] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:01:17.613231 4804 raft_consensus.cc:979] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45: : Instructing follower df655d53530449fc85de33de1c229815 to start an election
I20240502 14:01:17.613318 4804 raft_consensus.cc:1067] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 4 LEADER]: Signalling peer df655d53530449fc85de33de1c229815 to start an election
I20240502 14:01:17.614037 4478 raft_consensus.cc:3037] T c965f0e11bc1433f8ccdc20c98b4b4b9 P b3988f2f38654eedac11fa475e56e92f [term 3 LEARNER]: Advancing to term 4
I20240502 14:01:17.614521 4005 raft_consensus.cc:1261] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 4 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240502 14:01:17.615041 4822 consensus_queue.cc:1035] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:01:17.615080 4154 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c965f0e11bc1433f8ccdc20c98b4b4b9"
dest_uuid: "df655d53530449fc85de33de1c229815"
from {username='slave'} at 127.0.0.1:33578
I20240502 14:01:17.615108 4478 raft_consensus.cc:1261] T c965f0e11bc1433f8ccdc20c98b4b4b9 P b3988f2f38654eedac11fa475e56e92f [term 4 LEARNER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240502 14:01:17.615195 4154 raft_consensus.cc:482] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:17.615252 4154 raft_consensus.cc:3037] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:01:17.615479 4822 consensus_queue.cc:1035] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:01:17.616405 4154 raft_consensus.cc:504] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:17.616732 4150 raft_consensus.cc:1226] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 5 FOLLOWER]: Rejecting Update request from peer 24fd5d6c84c1483ea8f39bb975e26e45 for earlier term 4. Current term is 5. Ops: []
I20240502 14:01:17.616990 4000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c965f0e11bc1433f8ccdc20c98b4b4b9" candidate_uuid: "df655d53530449fc85de33de1c229815" candidate_term: 5 candidate_status { last_received { term: 4 index: 31 } } ignore_live_leader: true dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
I20240502 14:01:17.616744 4154 leader_election.cc:288] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [CANDIDATE]: Term 5 election: Requested vote from peers bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667), 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251)
I20240502 14:01:17.617408 4082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c965f0e11bc1433f8ccdc20c98b4b4b9" candidate_uuid: "df655d53530449fc85de33de1c229815" candidate_term: 5 candidate_status { last_received { term: 4 index: 31 } } ignore_live_leader: true dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
I20240502 14:01:17.617506 4082 raft_consensus.cc:3032] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 14:01:17.617542 4082 raft_consensus.cc:729] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Running, Role: LEADER
I20240502 14:01:17.617660 4082 consensus_queue.cc:260] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 31, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:17.617805 4082 raft_consensus.cc:3037] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:01:17.618682 4082 raft_consensus.cc:2454] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df655d53530449fc85de33de1c229815 in term 5.
I20240502 14:01:17.618903 4123 leader_election.cc:302] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:17.619066 4837 raft_consensus.cc:2781] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 14:01:17.619161 4837 raft_consensus.cc:686] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 5 LEADER]: Becoming Leader. State: Replica: df655d53530449fc85de33de1c229815, State: Running, Role: LEADER
I20240502 14:01:17.619282 4837 consensus_queue.cc:237] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.31, Last appended by leader: 31, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:17.620595 3894 catalog_manager.cc:5484] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 reported cstate change: term changed from 4 to 5, leader changed from 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66) to df655d53530449fc85de33de1c229815 (127.0.145.67). New cstate: current_term: 5 leader_uuid: "df655d53530449fc85de33de1c229815" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:17.671314 4858 raft_consensus.cc:1050] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815: attempting to promote NON_VOTER 5af426c57bc3434c9fbdcdff873a565c to VOTER
I20240502 14:01:17.671720 4858 consensus_queue.cc:237] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } }
I20240502 14:01:17.672888 4401 raft_consensus.cc:1261] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 5af426c57bc3434c9fbdcdff873a565c [term 1 LEARNER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 14:01:17.672953 4005 raft_consensus.cc:1261] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 14:01:17.673249 4688 consensus_queue.cc:1035] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:01:17.673580 4082 raft_consensus.cc:1261] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 14:01:17.674012 4860 consensus_queue.cc:1035] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:01:17.674336 4860 consensus_queue.cc:1035] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:01:17.675745 4688 raft_consensus.cc:2932] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } }
I20240502 14:01:17.676090 4401 raft_consensus.cc:2932] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 5af426c57bc3434c9fbdcdff873a565c [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } }
I20240502 14:01:17.676683 4082 raft_consensus.cc:2932] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } }
I20240502 14:01:17.677994 3894 catalog_manager.cc:5484] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 5af426c57bc3434c9fbdcdff873a565c reported cstate change: config changed from index 28 to 29, 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df655d53530449fc85de33de1c229815" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } }
I20240502 14:01:17.677999 4005 raft_consensus.cc:2932] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } }
I20240502 14:01:17.681828 4150 consensus_queue.cc:237] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } }
I20240502 14:01:17.682890 4082 raft_consensus.cc:1261] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 14:01:17.682967 4401 raft_consensus.cc:1261] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 5af426c57bc3434c9fbdcdff873a565c [term 1 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 14:01:17.683372 4860 consensus_queue.cc:1035] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:01:17.683588 4858 consensus_queue.cc:1035] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:01:17.685061 4837 raft_consensus.cc:2932] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } }
I20240502 14:01:17.685791 4401 raft_consensus.cc:2932] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 5af426c57bc3434c9fbdcdff873a565c [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } }
I20240502 14:01:17.685791 4082 raft_consensus.cc:2932] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } }
I20240502 14:01:17.687575 3894 catalog_manager.cc:5484] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 reported cstate change: config changed from index 29 to 30, VOTER bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65) evicted. New cstate: current_term: 1 leader_uuid: "df655d53530449fc85de33de1c229815" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:01:17.688410 3890 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7ac79a5b1f514bdcb36e0db4ff6e3a59 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:01:17.689637 3989 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ac79a5b1f514bdcb36e0db4ff6e3a59 with delete_type TABLET_DATA_TOMBSTONED (TS bd060dd4339f4fa5a985596bf7dec0ae not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:57056
I20240502 14:01:17.690169 4865 tablet_replica.cc:330] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P bd060dd4339f4fa5a985596bf7dec0ae: stopping tablet replica
I20240502 14:01:17.690311 4865 raft_consensus.cc:2229] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:17.690474 4865 raft_consensus.cc:2258] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:17.690991 4865 ts_tablet_manager.cc:1875] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P bd060dd4339f4fa5a985596bf7dec0ae: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:17.692960 4865 ts_tablet_manager.cc:1888] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P bd060dd4339f4fa5a985596bf7dec0ae: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20240502 14:01:17.693042 4865 log.cc:1196] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P bd060dd4339f4fa5a985596bf7dec0ae: Deleting WAL directory at /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-0-root/wals/7ac79a5b1f514bdcb36e0db4ff6e3a59
I20240502 14:01:17.693512 3889 catalog_manager.cc:4830] TS bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667): tablet 7ac79a5b1f514bdcb36e0db4ff6e3a59 (table test-workload [id=66735bdbe8fc4765af013422ce730ad0]) successfully deleted
I20240502 14:01:17.757622 4860 raft_consensus.cc:1050] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815: attempting to promote NON_VOTER e1de31532912433dbed75c2bb01bc42a to VOTER
I20240502 14:01:17.758025 4860 consensus_queue.cc:237] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 27, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } }
I20240502 14:01:17.759275 4005 raft_consensus.cc:1261] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 14:01:17.759315 4553 raft_consensus.cc:1261] T fb4c3dcde8764e17bb084cb37fef706f P e1de31532912433dbed75c2bb01bc42a [term 2 LEARNER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 14:01:17.759517 4082 raft_consensus.cc:1261] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 14:01:17.759578 4688 consensus_queue.cc:1035] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:01:17.759708 4858 consensus_queue.cc:1035] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:01:17.760023 4858 consensus_queue.cc:1035] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:01:17.761607 4860 raft_consensus.cc:2932] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, e1de31532912433dbed75c2bb01bc42a (127.0.145.70) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:17.761972 4005 raft_consensus.cc:2932] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, e1de31532912433dbed75c2bb01bc42a (127.0.145.70) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:17.762476 4553 raft_consensus.cc:2932] T fb4c3dcde8764e17bb084cb37fef706f P e1de31532912433dbed75c2bb01bc42a [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, e1de31532912433dbed75c2bb01bc42a (127.0.145.70) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:17.763758 4082 raft_consensus.cc:2932] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, e1de31532912433dbed75c2bb01bc42a (127.0.145.70) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:17.764252 3903 catalog_manager.cc:5484] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae reported cstate change: config changed from index 29 to 30, e1de31532912433dbed75c2bb01bc42a (127.0.145.70) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "df655d53530449fc85de33de1c229815" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:17.767721 4150 consensus_queue.cc:237] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } }
I20240502 14:01:17.768659 4082 raft_consensus.cc:1261] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 14:01:17.768723 4553 raft_consensus.cc:1261] T fb4c3dcde8764e17bb084cb37fef706f P e1de31532912433dbed75c2bb01bc42a [term 2 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 14:01:17.769053 4837 consensus_queue.cc:1035] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:01:17.769217 4688 consensus_queue.cc:1035] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:01:17.771595 4858 raft_consensus.cc:2932] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:17.773010 4812 raft_consensus.cc:1050] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae: attempting to promote NON_VOTER e1de31532912433dbed75c2bb01bc42a to VOTER
I20240502 14:01:17.773156 3890 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fb4c3dcde8764e17bb084cb37fef706f with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:01:17.773080 4082 raft_consensus.cc:2932] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:17.773345 4812 consensus_queue.cc:237] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 28, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } }
I20240502 14:01:17.773488 4553 raft_consensus.cc:2932] T fb4c3dcde8764e17bb084cb37fef706f P e1de31532912433dbed75c2bb01bc42a [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:17.774874 4154 raft_consensus.cc:1261] T 46b4a50dee854f62aac3a02889525b40 P df655d53530449fc85de33de1c229815 [term 2 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 14:01:17.774852 3894 catalog_manager.cc:5484] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 reported cstate change: config changed from index 30 to 31, VOTER bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65) evicted. New cstate: current_term: 2 leader_uuid: "df655d53530449fc85de33de1c229815" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:01:17.774964 4082 raft_consensus.cc:1261] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 14:01:17.775249 4813 consensus_queue.cc:1035] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:01:17.775508 4552 raft_consensus.cc:1261] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a [term 2 LEARNER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 14:01:17.775882 4812 consensus_queue.cc:1035] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:01:17.776187 4812 consensus_queue.cc:1035] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:01:17.777513 4812 raft_consensus.cc:2932] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 28 to 30, e1de31532912433dbed75c2bb01bc42a (127.0.145.70) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:17.779104 4154 raft_consensus.cc:2932] T 46b4a50dee854f62aac3a02889525b40 P df655d53530449fc85de33de1c229815 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, e1de31532912433dbed75c2bb01bc42a (127.0.145.70) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:17.779517 4078 raft_consensus.cc:2932] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, e1de31532912433dbed75c2bb01bc42a (127.0.145.70) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:17.779551 4552 raft_consensus.cc:2932] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, e1de31532912433dbed75c2bb01bc42a (127.0.145.70) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:17.779664 3900 catalog_manager.cc:5484] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae reported cstate change: config changed from index 28 to 30, e1de31532912433dbed75c2bb01bc42a (127.0.145.70) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:01:17.783265 3989 tablet_service.cc:1514] Processing DeleteTablet for tablet fb4c3dcde8764e17bb084cb37fef706f with delete_type TABLET_DATA_TOMBSTONED (TS bd060dd4339f4fa5a985596bf7dec0ae not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:57056
I20240502 14:01:17.783531 4865 tablet_replica.cc:330] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae: stopping tablet replica
I20240502 14:01:17.783658 4865 raft_consensus.cc:2229] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:17.783782 4865 raft_consensus.cc:2258] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:17.784267 4005 consensus_queue.cc:237] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } }
I20240502 14:01:17.784510 4865 ts_tablet_manager.cc:1875] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:17.785676 4078 raft_consensus.cc:1261] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 14:01:17.785677 4553 raft_consensus.cc:1261] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a [term 2 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 14:01:17.786245 4875 consensus_queue.cc:1035] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:01:17.786440 4872 consensus_queue.cc:1035] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:01:17.786491 4865 ts_tablet_manager.cc:1888] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.30
I20240502 14:01:17.786571 4865 log.cc:1196] T fb4c3dcde8764e17bb084cb37fef706f P bd060dd4339f4fa5a985596bf7dec0ae: Deleting WAL directory at /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-0-root/wals/fb4c3dcde8764e17bb084cb37fef706f
I20240502 14:01:17.787065 3889 catalog_manager.cc:4830] TS bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667): tablet fb4c3dcde8764e17bb084cb37fef706f (table test-workload [id=66735bdbe8fc4765af013422ce730ad0]) successfully deleted
I20240502 14:01:17.788002 4812 raft_consensus.cc:2932] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER df655d53530449fc85de33de1c229815 (127.0.145.67) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:17.788363 4078 raft_consensus.cc:2932] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER df655d53530449fc85de33de1c229815 (127.0.145.67) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:17.789005 3889 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 46b4a50dee854f62aac3a02889525b40 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:01:17.789901 3898 catalog_manager.cc:5484] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae reported cstate change: config changed from index 30 to 31, VOTER df655d53530449fc85de33de1c229815 (127.0.145.67) evicted. New cstate: current_term: 2 leader_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:01:17.790007 4553 raft_consensus.cc:2932] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER df655d53530449fc85de33de1c229815 (127.0.145.67) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:17.791788 4142 tablet_service.cc:1514] Processing DeleteTablet for tablet 46b4a50dee854f62aac3a02889525b40 with delete_type TABLET_DATA_TOMBSTONED (TS df655d53530449fc85de33de1c229815 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:33568
I20240502 14:01:17.791994 4851 tablet_replica.cc:330] T 46b4a50dee854f62aac3a02889525b40 P df655d53530449fc85de33de1c229815: stopping tablet replica
I20240502 14:01:17.792126 4851 raft_consensus.cc:2229] T 46b4a50dee854f62aac3a02889525b40 P df655d53530449fc85de33de1c229815 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:17.792275 4851 raft_consensus.cc:2258] T 46b4a50dee854f62aac3a02889525b40 P df655d53530449fc85de33de1c229815 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:17.792866 4851 ts_tablet_manager.cc:1875] T 46b4a50dee854f62aac3a02889525b40 P df655d53530449fc85de33de1c229815: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:17.794258 4851 ts_tablet_manager.cc:1888] T 46b4a50dee854f62aac3a02889525b40 P df655d53530449fc85de33de1c229815: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.30
I20240502 14:01:17.794332 4851 log.cc:1196] T 46b4a50dee854f62aac3a02889525b40 P df655d53530449fc85de33de1c229815: Deleting WAL directory at /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-2-root/wals/46b4a50dee854f62aac3a02889525b40
I20240502 14:01:17.794660 3890 catalog_manager.cc:4830] TS df655d53530449fc85de33de1c229815 (127.0.145.67:34309): tablet 46b4a50dee854f62aac3a02889525b40 (table test-workload [id=66735bdbe8fc4765af013422ce730ad0]) successfully deleted
I20240502 14:01:18.005700 4078 raft_consensus.cc:1261] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 5 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20240502 14:01:18.006078 4860 consensus_queue.cc:1035] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 14:01:18.008121 4478 raft_consensus.cc:3037] T c965f0e11bc1433f8ccdc20c98b4b4b9 P b3988f2f38654eedac11fa475e56e92f [term 4 LEARNER]: Advancing to term 5
I20240502 14:01:18.008121 4000 raft_consensus.cc:3037] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:01:18.009217 4000 raft_consensus.cc:1261] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 5 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20240502 14:01:18.009217 4478 raft_consensus.cc:1261] T c965f0e11bc1433f8ccdc20c98b4b4b9 P b3988f2f38654eedac11fa475e56e92f [term 5 LEARNER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20240502 14:01:18.009649 4858 consensus_queue.cc:1035] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 14:01:18.009888 4858 consensus_queue.cc:1035] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 14:01:18.011369 4860 raft_consensus.cc:1050] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815: attempting to promote NON_VOTER b3988f2f38654eedac11fa475e56e92f to VOTER
I20240502 14:01:18.011719 4860 consensus_queue.cc:237] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 5.32, Last appended by leader: 31, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
I20240502 14:01:18.012861 4000 raft_consensus.cc:1261] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 5 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20240502 14:01:18.012861 4078 raft_consensus.cc:1261] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 5 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20240502 14:01:18.012984 4478 raft_consensus.cc:1261] T c965f0e11bc1433f8ccdc20c98b4b4b9 P b3988f2f38654eedac11fa475e56e92f [term 5 LEARNER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20240502 14:01:18.013227 4688 consensus_queue.cc:1035] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 14:01:18.013375 4858 consensus_queue.cc:1035] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 14:01:18.013571 4688 consensus_queue.cc:1035] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 14:01:18.014623 4858 raft_consensus.cc:2932] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 5 LEADER]: Committing config change with OpId 5.33: config changed from index 29 to 33, b3988f2f38654eedac11fa475e56e92f (127.0.145.69) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:18.015002 4478 raft_consensus.cc:2932] T c965f0e11bc1433f8ccdc20c98b4b4b9 P b3988f2f38654eedac11fa475e56e92f [term 5 FOLLOWER]: Committing config change with OpId 5.33: config changed from index 29 to 33, b3988f2f38654eedac11fa475e56e92f (127.0.145.69) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:18.015280 4000 raft_consensus.cc:2932] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 5 FOLLOWER]: Committing config change with OpId 5.33: config changed from index 29 to 33, b3988f2f38654eedac11fa475e56e92f (127.0.145.69) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:18.016676 3903 catalog_manager.cc:5484] T c965f0e11bc1433f8ccdc20c98b4b4b9 P b3988f2f38654eedac11fa475e56e92f reported cstate change: config changed from index 29 to 33, b3988f2f38654eedac11fa475e56e92f (127.0.145.69) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "df655d53530449fc85de33de1c229815" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:18.017261 4078 raft_consensus.cc:2932] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 5 FOLLOWER]: Committing config change with OpId 5.33: config changed from index 29 to 33, b3988f2f38654eedac11fa475e56e92f (127.0.145.69) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:18.019780 4154 consensus_queue.cc:237] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 5.33, Last appended by leader: 31, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
I20240502 14:01:18.020807 4000 raft_consensus.cc:1261] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 5 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 5 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20240502 14:01:18.020808 4478 raft_consensus.cc:1261] T c965f0e11bc1433f8ccdc20c98b4b4b9 P b3988f2f38654eedac11fa475e56e92f [term 5 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 5 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20240502 14:01:18.021200 4837 consensus_queue.cc:1035] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 14:01:18.021555 4837 consensus_queue.cc:1035] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 14:01:18.022547 4688 raft_consensus.cc:2932] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 5 LEADER]: Committing config change with OpId 5.34: config changed from index 33 to 34, VOTER 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:18.023078 4000 raft_consensus.cc:2932] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 5 FOLLOWER]: Committing config change with OpId 5.34: config changed from index 33 to 34, VOTER 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:18.023078 4478 raft_consensus.cc:2932] T c965f0e11bc1433f8ccdc20c98b4b4b9 P b3988f2f38654eedac11fa475e56e92f [term 5 FOLLOWER]: Committing config change with OpId 5.34: config changed from index 33 to 34, VOTER 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:18.023640 3890 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c965f0e11bc1433f8ccdc20c98b4b4b9 with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:01:18.024549 3900 catalog_manager.cc:5484] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 reported cstate change: config changed from index 33 to 34, VOTER 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66) evicted. New cstate: current_term: 5 leader_uuid: "df655d53530449fc85de33de1c229815" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:01:18.026751 4067 tablet_service.cc:1514] Processing DeleteTablet for tablet c965f0e11bc1433f8ccdc20c98b4b4b9 with delete_type TABLET_DATA_TOMBSTONED (TS 24fd5d6c84c1483ea8f39bb975e26e45 not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:59772
I20240502 14:01:18.027182 4876 tablet_replica.cc:330] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45: stopping tablet replica
I20240502 14:01:18.027333 4876 raft_consensus.cc:2229] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:18.027571 4876 raft_consensus.cc:2258] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:18.028084 4876 ts_tablet_manager.cc:1875] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:18.030148 4876 ts_tablet_manager.cc:1888] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 5.33
I20240502 14:01:18.030222 4876 log.cc:1196] T c965f0e11bc1433f8ccdc20c98b4b4b9 P 24fd5d6c84c1483ea8f39bb975e26e45: Deleting WAL directory at /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-1-root/wals/c965f0e11bc1433f8ccdc20c98b4b4b9
I20240502 14:01:18.030769 3890 catalog_manager.cc:4830] TS 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251): tablet c965f0e11bc1433f8ccdc20c98b4b4b9 (table test-workload [id=66735bdbe8fc4765af013422ce730ad0]) successfully deleted
I20240502 14:01:18.159327 4807 tablet_copy_client.cc:523] T 144739b71bf6474d90399816120b3b59 P b3988f2f38654eedac11fa475e56e92f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:18.160624 4807 tablet_bootstrap.cc:492] T 144739b71bf6474d90399816120b3b59 P b3988f2f38654eedac11fa475e56e92f: Bootstrap starting.
I20240502 14:01:18.178535 4807 tablet_bootstrap.cc:492] T 144739b71bf6474d90399816120b3b59 P b3988f2f38654eedac11fa475e56e92f: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=106 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:18.178732 4807 tablet_bootstrap.cc:492] T 144739b71bf6474d90399816120b3b59 P b3988f2f38654eedac11fa475e56e92f: Bootstrap complete.
I20240502 14:01:18.178862 4807 ts_tablet_manager.cc:1367] T 144739b71bf6474d90399816120b3b59 P b3988f2f38654eedac11fa475e56e92f: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.004s
I20240502 14:01:18.179201 4807 raft_consensus.cc:348] T 144739b71bf6474d90399816120b3b59 P b3988f2f38654eedac11fa475e56e92f [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:18.179301 4807 raft_consensus.cc:729] T 144739b71bf6474d90399816120b3b59 P b3988f2f38654eedac11fa475e56e92f [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: b3988f2f38654eedac11fa475e56e92f, State: Initialized, Role: LEARNER
I20240502 14:01:18.179373 4807 consensus_queue.cc:260] T 144739b71bf6474d90399816120b3b59 P b3988f2f38654eedac11fa475e56e92f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 4.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:18.179620 4807 ts_tablet_manager.cc:1398] T 144739b71bf6474d90399816120b3b59 P b3988f2f38654eedac11fa475e56e92f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:18.180059 4016 tablet_copy_service.cc:342] P bd060dd4339f4fa5a985596bf7dec0ae: Request end of tablet copy session b3988f2f38654eedac11fa475e56e92f-144739b71bf6474d90399816120b3b59 received from {username='slave'} at 127.0.0.1:57094
I20240502 14:01:18.180135 4016 tablet_copy_service.cc:433] P bd060dd4339f4fa5a985596bf7dec0ae: ending tablet copy session b3988f2f38654eedac11fa475e56e92f-144739b71bf6474d90399816120b3b59 on tablet 144739b71bf6474d90399816120b3b59 with peer b3988f2f38654eedac11fa475e56e92f
I20240502 14:01:18.268999 4478 raft_consensus.cc:3037] T 144739b71bf6474d90399816120b3b59 P b3988f2f38654eedac11fa475e56e92f [term 4 LEARNER]: Advancing to term 5
I20240502 14:01:18.683809 3957 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:01:18.685465 4154 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "144739b71bf6474d90399816120b3b59"
dest_uuid: "df655d53530449fc85de33de1c229815"
mode: GRACEFUL
new_leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
from {username='slave'} at 127.0.0.1:33630
I20240502 14:01:18.685608 4154 raft_consensus.cc:595] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 5 LEADER]: Received request to transfer leadership to TS 24fd5d6c84c1483ea8f39bb975e26e45
I20240502 14:01:18.686264 4154 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7ac79a5b1f514bdcb36e0db4ff6e3a59"
dest_uuid: "df655d53530449fc85de33de1c229815"
mode: GRACEFUL
new_leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
from {username='slave'} at 127.0.0.1:33630
I20240502 14:01:18.686385 4154 raft_consensus.cc:595] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [term 1 LEADER]: Received request to transfer leadership to TS 24fd5d6c84c1483ea8f39bb975e26e45
I20240502 14:01:18.686949 4078 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dba8942501c440b18f71c7106878d0c4"
dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
mode: GRACEFUL
new_leader_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
from {username='slave'} at 127.0.0.1:59812
I20240502 14:01:18.687047 4078 raft_consensus.cc:595] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Received request to transfer leadership to TS bd060dd4339f4fa5a985596bf7dec0ae
I20240502 14:01:18.687338 3957 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240502 14:01:18.687458 3957 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:01:18.761752 4858 raft_consensus.cc:979] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815: : Instructing follower 24fd5d6c84c1483ea8f39bb975e26e45 to start an election
I20240502 14:01:18.761849 4858 raft_consensus.cc:1067] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [term 1 LEADER]: Signalling peer 24fd5d6c84c1483ea8f39bb975e26e45 to start an election
I20240502 14:01:18.762203 4078 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7ac79a5b1f514bdcb36e0db4ff6e3a59"
dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
from {username='slave'} at 127.0.0.1:59788
I20240502 14:01:18.762312 4078 raft_consensus.cc:482] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:18.762357 4078 raft_consensus.cc:3037] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:18.763114 4078 raft_consensus.cc:504] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } }
I20240502 14:01:18.763411 4078 leader_election.cc:288] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 2 election: Requested vote from peers df655d53530449fc85de33de1c229815 (127.0.145.67:34309), 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68:34601)
I20240502 14:01:18.763696 4154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ac79a5b1f514bdcb36e0db4ff6e3a59" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "df655d53530449fc85de33de1c229815"
I20240502 14:01:18.763738 4401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ac79a5b1f514bdcb36e0db4ff6e3a59" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "5af426c57bc3434c9fbdcdff873a565c"
I20240502 14:01:18.763789 4154 raft_consensus.cc:3032] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:01:18.763870 4401 raft_consensus.cc:3037] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 5af426c57bc3434c9fbdcdff873a565c [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:18.763876 4154 raft_consensus.cc:729] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df655d53530449fc85de33de1c229815, State: Running, Role: LEADER
I20240502 14:01:18.764024 4154 consensus_queue.cc:260] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } }
I20240502 14:01:18.764153 4154 raft_consensus.cc:3037] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:18.764660 4401 raft_consensus.cc:2454] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 5af426c57bc3434c9fbdcdff873a565c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 2.
I20240502 14:01:18.764843 4154 raft_consensus.cc:2454] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 2.
I20240502 14:01:18.764956 4048 leader_election.cc:302] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, 5af426c57bc3434c9fbdcdff873a565c; no voters:
I20240502 14:01:18.765110 4855 raft_consensus.cc:2781] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:18.765265 4855 raft_consensus.cc:686] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 LEADER]: Becoming Leader. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Running, Role: LEADER
I20240502 14:01:18.765394 4855 consensus_queue.cc:237] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } }
I20240502 14:01:18.766729 3898 catalog_manager.cc:5484] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 24fd5d6c84c1483ea8f39bb975e26e45 reported cstate change: term changed from 1 to 2, leader changed from df655d53530449fc85de33de1c229815 (127.0.145.67) to 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66). New cstate: current_term: 2 leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:18.792973 4154 consensus_queue.cc:237] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:18.794152 4078 raft_consensus.cc:1261] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
W20240502 14:01:18.794238 4120 consensus_peers.cc:485] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 -> Peer b3988f2f38654eedac11fa475e56e92f (127.0.145.69:45021): Couldn't send request to peer b3988f2f38654eedac11fa475e56e92f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb4c3dcde8764e17bb084cb37fef706f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:18.794152 4553 raft_consensus.cc:1261] T fb4c3dcde8764e17bb084cb37fef706f P e1de31532912433dbed75c2bb01bc42a [term 2 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240502 14:01:18.794538 4858 consensus_queue.cc:1035] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 14:01:18.794719 4879 consensus_queue.cc:1035] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 14:01:18.796079 4858 raft_consensus.cc:2932] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, NON_VOTER b3988f2f38654eedac11fa475e56e92f (127.0.145.69) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } } }
I20240502 14:01:18.796530 4553 raft_consensus.cc:2932] T fb4c3dcde8764e17bb084cb37fef706f P e1de31532912433dbed75c2bb01bc42a [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, NON_VOTER b3988f2f38654eedac11fa475e56e92f (127.0.145.69) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } } }
I20240502 14:01:18.797487 4078 raft_consensus.cc:2932] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, NON_VOTER b3988f2f38654eedac11fa475e56e92f (127.0.145.69) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } } }
I20240502 14:01:18.798305 3898 catalog_manager.cc:5484] T fb4c3dcde8764e17bb084cb37fef706f P e1de31532912433dbed75c2bb01bc42a reported cstate change: config changed from index 31 to 32, NON_VOTER b3988f2f38654eedac11fa475e56e92f (127.0.145.69) added. New cstate: current_term: 2 leader_uuid: "df655d53530449fc85de33de1c229815" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } } }
W20240502 14:01:18.798667 3952 auto_rebalancer.cc:253] failed to send replica move request: Remote error: Service unavailable: leader transfer in progress
W20240502 14:01:18.800504 3952 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 144739b71bf6474d90399816120b3b59, TS 24fd5d6c84c1483ea8f39bb975e26e45 -> TS 5af426c57bc3434c9fbdcdff873a565c move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
W20240502 14:01:18.800558 3952 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 144739b71bf6474d90399816120b3b59, TS 24fd5d6c84c1483ea8f39bb975e26e45 -> TS 5af426c57bc3434c9fbdcdff873a565c move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
W20240502 14:01:18.802103 3952 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet c965f0e11bc1433f8ccdc20c98b4b4b9, TS bd060dd4339f4fa5a985596bf7dec0ae -> TS e1de31532912433dbed75c2bb01bc42a move failed, destination replica disappeared from tablet's Raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
W20240502 14:01:18.802158 3952 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet c965f0e11bc1433f8ccdc20c98b4b4b9, TS bd060dd4339f4fa5a985596bf7dec0ae -> TS e1de31532912433dbed75c2bb01bc42a move failed, destination replica disappeared from tablet's Raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
W20240502 14:01:18.806501 3952 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 3d0d4aeaec264d04bae2c34e6edc6ec7, TS df655d53530449fc85de33de1c229815 -> TS b3988f2f38654eedac11fa475e56e92f move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } }
W20240502 14:01:18.806547 3952 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 3d0d4aeaec264d04bae2c34e6edc6ec7, TS df655d53530449fc85de33de1c229815 -> TS b3988f2f38654eedac11fa475e56e92f move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } }
W20240502 14:01:18.807947 3952 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 46b4a50dee854f62aac3a02889525b40, TS 24fd5d6c84c1483ea8f39bb975e26e45 -> TS 5af426c57bc3434c9fbdcdff873a565c move failed, destination replica disappeared from tablet's Raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } }
W20240502 14:01:18.807989 3952 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 46b4a50dee854f62aac3a02889525b40, TS 24fd5d6c84c1483ea8f39bb975e26e45 -> TS 5af426c57bc3434c9fbdcdff873a565c move failed, destination replica disappeared from tablet's Raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } }
W20240502 14:01:18.809355 3952 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 975becfb8df0430fa8c748a58dcbcfb1, TS bd060dd4339f4fa5a985596bf7dec0ae -> TS b3988f2f38654eedac11fa475e56e92f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
W20240502 14:01:18.809398 3952 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 975becfb8df0430fa8c748a58dcbcfb1, TS bd060dd4339f4fa5a985596bf7dec0ae -> TS b3988f2f38654eedac11fa475e56e92f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:18.843299 4879 raft_consensus.cc:1050] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815: attempting to promote NON_VOTER b3988f2f38654eedac11fa475e56e92f to VOTER
W20240502 14:01:18.843387 4879 raft_consensus.cc:1053] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815: Unable to promote non-voter b3988f2f38654eedac11fa475e56e92f: Service unavailable: leader transfer in progress
I20240502 14:01:18.940697 4879 raft_consensus.cc:979] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815: : Instructing follower 24fd5d6c84c1483ea8f39bb975e26e45 to start an election
I20240502 14:01:18.940850 4886 raft_consensus.cc:1067] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 5 LEADER]: Signalling peer 24fd5d6c84c1483ea8f39bb975e26e45 to start an election
I20240502 14:01:18.941216 4078 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "144739b71bf6474d90399816120b3b59"
dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
from {username='slave'} at 127.0.0.1:59788
I20240502 14:01:18.941335 4078 raft_consensus.cc:482] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:18.941398 4078 raft_consensus.cc:3037] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 5 FOLLOWER]: Advancing to term 6
I20240502 14:01:18.942299 4078 raft_consensus.cc:504] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:18.942631 4078 leader_election.cc:288] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 6 election: Requested vote from peers bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:18.942924 4000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144739b71bf6474d90399816120b3b59" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 6 candidate_status { last_received { term: 5 index: 32 } } ignore_live_leader: true dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
I20240502 14:01:18.943017 4000 raft_consensus.cc:3037] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 5 FOLLOWER]: Advancing to term 6
I20240502 14:01:18.942983 4150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144739b71bf6474d90399816120b3b59" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 6 candidate_status { last_received { term: 5 index: 32 } } ignore_live_leader: true dest_uuid: "df655d53530449fc85de33de1c229815"
I20240502 14:01:18.943085 4150 raft_consensus.cc:3032] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 14:01:18.943112 4150 raft_consensus.cc:729] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df655d53530449fc85de33de1c229815, State: Running, Role: LEADER
I20240502 14:01:18.943223 4150 consensus_queue.cc:260] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 5.32, Last appended by leader: 32, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:18.943389 4150 raft_consensus.cc:3037] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 5 FOLLOWER]: Advancing to term 6
I20240502 14:01:18.943905 4000 raft_consensus.cc:2454] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 6.
I20240502 14:01:18.944136 4047 leader_election.cc:302] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, bd060dd4339f4fa5a985596bf7dec0ae; no voters:
I20240502 14:01:18.944136 4150 raft_consensus.cc:2454] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 6.
I20240502 14:01:18.944334 4855 raft_consensus.cc:2781] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 6 FOLLOWER]: Leader election won for term 6
I20240502 14:01:18.944425 4855 raft_consensus.cc:686] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 6 LEADER]: Becoming Leader. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Running, Role: LEADER
I20240502 14:01:18.944550 4855 consensus_queue.cc:237] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 5.32, Last appended by leader: 32, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:18.946120 3900 catalog_manager.cc:5484] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 reported cstate change: term changed from 5 to 6, leader changed from df655d53530449fc85de33de1c229815 (127.0.145.67) to 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66). New cstate: current_term: 6 leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:19.214298 4880 raft_consensus.cc:979] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45: : Instructing follower bd060dd4339f4fa5a985596bf7dec0ae to start an election
I20240502 14:01:19.214443 4855 raft_consensus.cc:1067] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Signalling peer bd060dd4339f4fa5a985596bf7dec0ae to start an election
I20240502 14:01:19.214848 4000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dba8942501c440b18f71c7106878d0c4"
dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
from {username='slave'} at 127.0.0.1:57072
I20240502 14:01:19.214965 4000 raft_consensus.cc:482] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:19.215023 4000 raft_consensus.cc:3037] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:19.215782 4000 raft_consensus.cc:504] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } }
I20240502 14:01:19.216105 4000 leader_election.cc:288] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 2 election: Requested vote from peers 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251), e1de31532912433dbed75c2bb01bc42a (127.0.145.70:41637)
I20240502 14:01:19.216382 4078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba8942501c440b18f71c7106878d0c4" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
I20240502 14:01:19.216390 4553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba8942501c440b18f71c7106878d0c4" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "e1de31532912433dbed75c2bb01bc42a"
I20240502 14:01:19.216516 4078 raft_consensus.cc:3032] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:01:19.216545 4553 raft_consensus.cc:3037] T dba8942501c440b18f71c7106878d0c4 P e1de31532912433dbed75c2bb01bc42a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:19.216564 4078 raft_consensus.cc:729] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Running, Role: LEADER
I20240502 14:01:19.216689 4078 consensus_queue.cc:260] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } }
I20240502 14:01:19.216850 4078 raft_consensus.cc:3037] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:19.217391 4553 raft_consensus.cc:2454] T dba8942501c440b18f71c7106878d0c4 P e1de31532912433dbed75c2bb01bc42a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 2.
I20240502 14:01:19.217541 4078 raft_consensus.cc:2454] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 2.
I20240502 14:01:19.217736 3969 leader_election.cc:302] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd060dd4339f4fa5a985596bf7dec0ae, e1de31532912433dbed75c2bb01bc42a; no voters:
I20240502 14:01:19.217887 4875 raft_consensus.cc:2781] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:19.218057 4875 raft_consensus.cc:686] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 LEADER]: Becoming Leader. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Running, Role: LEADER
I20240502 14:01:19.218191 4875 consensus_queue.cc:237] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } }
I20240502 14:01:19.219583 3900 catalog_manager.cc:5484] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae reported cstate change: term changed from 1 to 2, leader changed from 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66) to bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65). New cstate: current_term: 2 leader_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:19.315580 4894 ts_tablet_manager.cc:902] T fb4c3dcde8764e17bb084cb37fef706f P b3988f2f38654eedac11fa475e56e92f: Initiating tablet copy from peer df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:19.315580 4401 raft_consensus.cc:1261] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 5af426c57bc3434c9fbdcdff873a565c [term 2 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 14:01:19.316082 4855 consensus_queue.cc:1035] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:01:19.316160 4894 tablet_copy_client.cc:308] T fb4c3dcde8764e17bb084cb37fef706f P b3988f2f38654eedac11fa475e56e92f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.67:34309
I20240502 14:01:19.316524 4167 tablet_copy_service.cc:140] P df655d53530449fc85de33de1c229815: Received BeginTabletCopySession request for tablet fb4c3dcde8764e17bb084cb37fef706f from peer b3988f2f38654eedac11fa475e56e92f ({username='slave'} at 127.0.0.1:33622)
I20240502 14:01:19.316612 4167 tablet_copy_service.cc:161] P df655d53530449fc85de33de1c229815: Beginning new tablet copy session on tablet fb4c3dcde8764e17bb084cb37fef706f from peer b3988f2f38654eedac11fa475e56e92f at {username='slave'} at 127.0.0.1:33622: session id = b3988f2f38654eedac11fa475e56e92f-fb4c3dcde8764e17bb084cb37fef706f
I20240502 14:01:19.317646 4167 tablet_copy_source_session.cc:205] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:19.318743 4894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb4c3dcde8764e17bb084cb37fef706f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:19.319046 4154 raft_consensus.cc:1261] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P df655d53530449fc85de33de1c229815 [term 2 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 14:01:19.319380 4855 consensus_queue.cc:1035] T 7ac79a5b1f514bdcb36e0db4ff6e3a59 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:01:19.321105 4894 tablet_copy_client.cc:783] T fb4c3dcde8764e17bb084cb37fef706f P b3988f2f38654eedac11fa475e56e92f: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:19.321220 4894 tablet_copy_client.cc:647] T fb4c3dcde8764e17bb084cb37fef706f P b3988f2f38654eedac11fa475e56e92f: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:19.322124 4898 tablet_copy_client.cc:1002] T fb4c3dcde8764e17bb084cb37fef706f P b3988f2f38654eedac11fa475e56e92f: tablet copy: Injecting latency into file download: 2000
I20240502 14:01:19.446981 4478 raft_consensus.cc:3037] T 144739b71bf6474d90399816120b3b59 P b3988f2f38654eedac11fa475e56e92f [term 5 LEARNER]: Advancing to term 6
I20240502 14:01:19.448122 4478 raft_consensus.cc:1261] T 144739b71bf6474d90399816120b3b59 P b3988f2f38654eedac11fa475e56e92f [term 6 LEARNER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 6 index: 33. (index mismatch)
I20240502 14:01:19.448446 4855 consensus_queue.cc:1035] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 14:01:19.449952 4855 raft_consensus.cc:1050] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45: attempting to promote NON_VOTER b3988f2f38654eedac11fa475e56e92f to VOTER
W20240502 14:01:19.450045 4855 raft_consensus.cc:1053] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45: Unable to promote non-voter b3988f2f38654eedac11fa475e56e92f: Illegal state: Leader has not yet committed an operation in its own term
I20240502 14:01:19.487567 4000 raft_consensus.cc:1261] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 6 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 6 index: 33. (index mismatch)
I20240502 14:01:19.487893 4880 consensus_queue.cc:1035] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 14:01:19.490492 4150 raft_consensus.cc:1261] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 6 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 6 index: 33. (index mismatch)
I20240502 14:01:19.490844 4880 raft_consensus.cc:1050] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45: attempting to promote NON_VOTER b3988f2f38654eedac11fa475e56e92f to VOTER
I20240502 14:01:19.491215 4880 consensus_queue.cc:237] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 6.33, Last appended by leader: 32, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
I20240502 14:01:19.492750 4478 raft_consensus.cc:1261] T 144739b71bf6474d90399816120b3b59 P b3988f2f38654eedac11fa475e56e92f [term 6 LEARNER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 6 index: 33. Preceding OpId from leader: term: 6 index: 34. (index mismatch)
I20240502 14:01:19.493016 4154 raft_consensus.cc:1261] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 6 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 6 index: 34. (index mismatch)
I20240502 14:01:19.493016 4000 raft_consensus.cc:1261] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 6 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 6 index: 33. Preceding OpId from leader: term: 6 index: 34. (index mismatch)
I20240502 14:01:19.493114 4910 consensus_queue.cc:1035] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 14:01:19.493332 4906 consensus_queue.cc:1035] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 14:01:19.493598 4910 consensus_queue.cc:1035] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 14:01:19.495996 4000 raft_consensus.cc:2932] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 6 FOLLOWER]: Committing config change with OpId 6.34: config changed from index 29 to 34, b3988f2f38654eedac11fa475e56e92f (127.0.145.69) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:19.496222 4150 raft_consensus.cc:2932] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 6 FOLLOWER]: Committing config change with OpId 6.34: config changed from index 29 to 34, b3988f2f38654eedac11fa475e56e92f (127.0.145.69) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:19.496372 4880 raft_consensus.cc:2932] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 6 LEADER]: Committing config change with OpId 6.34: config changed from index 29 to 34, b3988f2f38654eedac11fa475e56e92f (127.0.145.69) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:19.498137 4478 raft_consensus.cc:2932] T 144739b71bf6474d90399816120b3b59 P b3988f2f38654eedac11fa475e56e92f [term 6 FOLLOWER]: Committing config change with OpId 6.34: config changed from index 29 to 34, b3988f2f38654eedac11fa475e56e92f (127.0.145.69) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:19.498279 3898 catalog_manager.cc:5484] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae reported cstate change: config changed from index 29 to 34, b3988f2f38654eedac11fa475e56e92f (127.0.145.69) changed from NON_VOTER to VOTER. New cstate: current_term: 6 leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:19.748898 4078 raft_consensus.cc:1261] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 14:01:19.749259 4875 consensus_queue.cc:1035] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:01:19.751622 4553 raft_consensus.cc:1261] T dba8942501c440b18f71c7106878d0c4 P e1de31532912433dbed75c2bb01bc42a [term 2 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 14:01:19.751993 4914 consensus_queue.cc:1035] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:01:20.687690 3957 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:01:20.689561 4078 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "144739b71bf6474d90399816120b3b59"
dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
mode: GRACEFUL
new_leader_uuid: "df655d53530449fc85de33de1c229815"
from {username='slave'} at 127.0.0.1:59812
I20240502 14:01:20.689683 4078 raft_consensus.cc:595] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 6 LEADER]: Received request to transfer leadership to TS df655d53530449fc85de33de1c229815
I20240502 14:01:20.690282 4078 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7908edf8bbe8419ab21d0c93cd2a3b35"
dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
mode: GRACEFUL
new_leader_uuid: "b3988f2f38654eedac11fa475e56e92f"
from {username='slave'} at 127.0.0.1:59812
I20240502 14:01:20.690395 4078 raft_consensus.cc:595] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 LEADER]: Received request to transfer leadership to TS b3988f2f38654eedac11fa475e56e92f
I20240502 14:01:20.691069 4000 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dba8942501c440b18f71c7106878d0c4"
dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
mode: GRACEFUL
new_leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
from {username='slave'} at 127.0.0.1:57080
I20240502 14:01:20.691196 4000 raft_consensus.cc:595] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 LEADER]: Received request to transfer leadership to TS 24fd5d6c84c1483ea8f39bb975e26e45
I20240502 14:01:20.691578 3957 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240502 14:01:20.691690 3957 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:01:20.717626 4919 raft_consensus.cc:979] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae: : Instructing follower 24fd5d6c84c1483ea8f39bb975e26e45 to start an election
I20240502 14:01:20.717725 4919 raft_consensus.cc:1067] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 LEADER]: Signalling peer 24fd5d6c84c1483ea8f39bb975e26e45 to start an election
I20240502 14:01:20.718046 4078 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dba8942501c440b18f71c7106878d0c4"
dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
from {username='slave'} at 127.0.0.1:59794
I20240502 14:01:20.718166 4078 raft_consensus.cc:482] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:20.718230 4078 raft_consensus.cc:3037] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:01:20.719228 4078 raft_consensus.cc:504] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } }
I20240502 14:01:20.719597 4078 leader_election.cc:288] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 3 election: Requested vote from peers bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667), e1de31532912433dbed75c2bb01bc42a (127.0.145.70:41637)
I20240502 14:01:20.719801 4000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba8942501c440b18f71c7106878d0c4" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 3 candidate_status { last_received { term: 2 index: 31 } } ignore_live_leader: true dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
I20240502 14:01:20.719890 4553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba8942501c440b18f71c7106878d0c4" candidate_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" candidate_term: 3 candidate_status { last_received { term: 2 index: 31 } } ignore_live_leader: true dest_uuid: "e1de31532912433dbed75c2bb01bc42a"
I20240502 14:01:20.720011 4553 raft_consensus.cc:3037] T dba8942501c440b18f71c7106878d0c4 P e1de31532912433dbed75c2bb01bc42a [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:01:20.720894 4553 raft_consensus.cc:2454] T dba8942501c440b18f71c7106878d0c4 P e1de31532912433dbed75c2bb01bc42a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 3.
I20240502 14:01:20.719928 4000 raft_consensus.cc:3032] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:01:20.721063 4000 raft_consensus.cc:729] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Running, Role: LEADER
I20240502 14:01:20.721130 4045 leader_election.cc:302] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, e1de31532912433dbed75c2bb01bc42a; no voters:
I20240502 14:01:20.721196 4000 consensus_queue.cc:260] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } }
I20240502 14:01:20.721292 4906 raft_consensus.cc:2781] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:01:20.721343 4000 raft_consensus.cc:3037] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:01:20.721452 4906 raft_consensus.cc:686] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 LEADER]: Becoming Leader. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Running, Role: LEADER
I20240502 14:01:20.721572 4906 consensus_queue.cc:237] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } }
I20240502 14:01:20.722141 4000 raft_consensus.cc:2454] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24fd5d6c84c1483ea8f39bb975e26e45 in term 3.
I20240502 14:01:20.722892 3900 catalog_manager.cc:5484] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 reported cstate change: term changed from 2 to 3, leader changed from bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65) to 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66). New cstate: current_term: 3 leader_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:20.753639 4920 raft_consensus.cc:979] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45: : Instructing follower df655d53530449fc85de33de1c229815 to start an election
I20240502 14:01:20.753759 4920 raft_consensus.cc:1067] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 6 LEADER]: Signalling peer df655d53530449fc85de33de1c229815 to start an election
I20240502 14:01:20.754144 4150 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "144739b71bf6474d90399816120b3b59"
dest_uuid: "df655d53530449fc85de33de1c229815"
from {username='slave'} at 127.0.0.1:33578
I20240502 14:01:20.754261 4150 raft_consensus.cc:482] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:20.754360 4150 raft_consensus.cc:3037] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 6 FOLLOWER]: Advancing to term 7
I20240502 14:01:20.755223 4150 raft_consensus.cc:504] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
I20240502 14:01:20.755810 4000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144739b71bf6474d90399816120b3b59" candidate_uuid: "df655d53530449fc85de33de1c229815" candidate_term: 7 candidate_status { last_received { term: 6 index: 34 } } ignore_live_leader: true dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
I20240502 14:01:20.755897 4150 leader_election.cc:288] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [CANDIDATE]: Term 7 election: Requested vote from peers bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667), 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251), b3988f2f38654eedac11fa475e56e92f (127.0.145.69:45021)
I20240502 14:01:20.755811 4078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144739b71bf6474d90399816120b3b59" candidate_uuid: "df655d53530449fc85de33de1c229815" candidate_term: 7 candidate_status { last_received { term: 6 index: 34 } } ignore_live_leader: true dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
I20240502 14:01:20.756008 4078 raft_consensus.cc:3032] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 6 LEADER]: Stepping down as leader of term 6
I20240502 14:01:20.756053 4078 raft_consensus.cc:729] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Running, Role: LEADER
I20240502 14:01:20.756186 4078 consensus_queue.cc:260] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 6.34, Last appended by leader: 34, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
I20240502 14:01:20.756381 4078 raft_consensus.cc:3037] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 6 FOLLOWER]: Advancing to term 7
I20240502 14:01:20.756875 4478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144739b71bf6474d90399816120b3b59" candidate_uuid: "df655d53530449fc85de33de1c229815" candidate_term: 7 candidate_status { last_received { term: 6 index: 34 } } ignore_live_leader: true dest_uuid: "b3988f2f38654eedac11fa475e56e92f"
I20240502 14:01:20.756965 4478 raft_consensus.cc:3037] T 144739b71bf6474d90399816120b3b59 P b3988f2f38654eedac11fa475e56e92f [term 6 FOLLOWER]: Advancing to term 7
I20240502 14:01:20.757146 4078 raft_consensus.cc:2454] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df655d53530449fc85de33de1c229815 in term 7.
I20240502 14:01:20.757725 4478 raft_consensus.cc:2454] T 144739b71bf6474d90399816120b3b59 P b3988f2f38654eedac11fa475e56e92f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df655d53530449fc85de33de1c229815 in term 7.
I20240502 14:01:20.755944 4000 raft_consensus.cc:3037] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 6 FOLLOWER]: Advancing to term 7
I20240502 14:01:20.757978 4120 leader_election.cc:302] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, b3988f2f38654eedac11fa475e56e92f, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:20.758158 4886 raft_consensus.cc:2781] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 7 FOLLOWER]: Leader election won for term 7
I20240502 14:01:20.758301 4886 raft_consensus.cc:686] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 7 LEADER]: Becoming Leader. State: Replica: df655d53530449fc85de33de1c229815, State: Running, Role: LEADER
I20240502 14:01:20.758420 4886 consensus_queue.cc:237] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 6.34, Last appended by leader: 34, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
I20240502 14:01:20.758785 4000 raft_consensus.cc:2454] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df655d53530449fc85de33de1c229815 in term 7.
I20240502 14:01:20.759902 3900 catalog_manager.cc:5484] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 reported cstate change: term changed from 6 to 7, leader changed from 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66) to df655d53530449fc85de33de1c229815 (127.0.145.67). New cstate: current_term: 7 leader_uuid: "df655d53530449fc85de33de1c229815" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:21.064358 4906 raft_consensus.cc:979] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45: : Instructing follower b3988f2f38654eedac11fa475e56e92f to start an election
I20240502 14:01:21.064471 4906 raft_consensus.cc:1067] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 LEADER]: Signalling peer b3988f2f38654eedac11fa475e56e92f to start an election
I20240502 14:01:21.064834 4478 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7908edf8bbe8419ab21d0c93cd2a3b35"
dest_uuid: "b3988f2f38654eedac11fa475e56e92f"
from {username='slave'} at 127.0.0.1:51174
I20240502 14:01:21.064988 4478 raft_consensus.cc:482] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:21.065050 4478 raft_consensus.cc:3037] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:01:21.066100 4478 raft_consensus.cc:504] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
I20240502 14:01:21.066529 4478 leader_election.cc:288] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f [CANDIDATE]: Term 4 election: Requested vote from peers 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:21.067085 4154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7908edf8bbe8419ab21d0c93cd2a3b35" candidate_uuid: "b3988f2f38654eedac11fa475e56e92f" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: true dest_uuid: "df655d53530449fc85de33de1c229815"
I20240502 14:01:21.067291 4154 raft_consensus.cc:3037] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:01:21.068189 4154 raft_consensus.cc:2454] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3988f2f38654eedac11fa475e56e92f in term 4.
I20240502 14:01:21.068480 4443 leader_election.cc:302] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3988f2f38654eedac11fa475e56e92f, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:21.068655 4845 raft_consensus.cc:2781] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:01:21.068833 4845 raft_consensus.cc:686] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f [term 4 LEADER]: Becoming Leader. State: Replica: b3988f2f38654eedac11fa475e56e92f, State: Running, Role: LEADER
I20240502 14:01:21.068969 4845 consensus_queue.cc:237] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
I20240502 14:01:21.070303 3900 catalog_manager.cc:5484] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f reported cstate change: term changed from 3 to 4, leader changed from 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66) to b3988f2f38654eedac11fa475e56e92f (127.0.145.69). New cstate: current_term: 4 leader_uuid: "b3988f2f38654eedac11fa475e56e92f" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:01:21.071018 4078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7908edf8bbe8419ab21d0c93cd2a3b35" candidate_uuid: "b3988f2f38654eedac11fa475e56e92f" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: true dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
I20240502 14:01:21.071123 4078 raft_consensus.cc:3032] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:01:21.071169 4078 raft_consensus.cc:729] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Running, Role: LEADER
I20240502 14:01:21.071292 4078 consensus_queue.cc:260] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
I20240502 14:01:21.071450 4078 raft_consensus.cc:3037] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:01:21.072278 4078 raft_consensus.cc:2454] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3988f2f38654eedac11fa475e56e92f in term 4.
I20240502 14:01:21.103258 4553 raft_consensus.cc:1261] T dba8942501c440b18f71c7106878d0c4 P e1de31532912433dbed75c2bb01bc42a [term 3 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240502 14:01:21.103583 4906 consensus_queue.cc:1035] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 14:01:21.105818 4000 raft_consensus.cc:1261] T dba8942501c440b18f71c7106878d0c4 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 FOLLOWER]: Refusing update from remote peer 24fd5d6c84c1483ea8f39bb975e26e45: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240502 14:01:21.106153 4906 consensus_queue.cc:1035] T dba8942501c440b18f71c7106878d0c4 P 24fd5d6c84c1483ea8f39bb975e26e45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 14:01:21.159601 4078 raft_consensus.cc:1261] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 7 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 6 index: 34. Preceding OpId from leader: term: 7 index: 35. (index mismatch)
I20240502 14:01:21.159929 4886 consensus_queue.cc:1035] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240502 14:01:21.322492 4894 tablet_copy_client.cc:523] T fb4c3dcde8764e17bb084cb37fef706f P b3988f2f38654eedac11fa475e56e92f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:21.324050 4894 tablet_bootstrap.cc:492] T fb4c3dcde8764e17bb084cb37fef706f P b3988f2f38654eedac11fa475e56e92f: Bootstrap starting.
I20240502 14:01:21.346872 4894 tablet_bootstrap.cc:492] T fb4c3dcde8764e17bb084cb37fef706f P b3988f2f38654eedac11fa475e56e92f: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=114 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:21.347146 4894 tablet_bootstrap.cc:492] T fb4c3dcde8764e17bb084cb37fef706f P b3988f2f38654eedac11fa475e56e92f: Bootstrap complete.
I20240502 14:01:21.347337 4894 ts_tablet_manager.cc:1367] T fb4c3dcde8764e17bb084cb37fef706f P b3988f2f38654eedac11fa475e56e92f: Time spent bootstrapping tablet: real 0.023s user 0.018s sys 0.004s
I20240502 14:01:21.347757 4894 raft_consensus.cc:348] T fb4c3dcde8764e17bb084cb37fef706f P b3988f2f38654eedac11fa475e56e92f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:21.347865 4894 raft_consensus.cc:729] T fb4c3dcde8764e17bb084cb37fef706f P b3988f2f38654eedac11fa475e56e92f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b3988f2f38654eedac11fa475e56e92f, State: Initialized, Role: LEARNER
I20240502 14:01:21.347995 4894 consensus_queue.cc:260] T fb4c3dcde8764e17bb084cb37fef706f P b3988f2f38654eedac11fa475e56e92f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:21.348286 4894 ts_tablet_manager.cc:1398] T fb4c3dcde8764e17bb084cb37fef706f P b3988f2f38654eedac11fa475e56e92f: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 14:01:21.348680 4167 tablet_copy_service.cc:342] P df655d53530449fc85de33de1c229815: Request end of tablet copy session b3988f2f38654eedac11fa475e56e92f-fb4c3dcde8764e17bb084cb37fef706f received from {username='slave'} at 127.0.0.1:33622
I20240502 14:01:21.348779 4167 tablet_copy_service.cc:433] P df655d53530449fc85de33de1c229815: ending tablet copy session b3988f2f38654eedac11fa475e56e92f-fb4c3dcde8764e17bb084cb37fef706f on tablet fb4c3dcde8764e17bb084cb37fef706f with peer b3988f2f38654eedac11fa475e56e92f
I20240502 14:01:21.350931 4478 raft_consensus.cc:1261] T 144739b71bf6474d90399816120b3b59 P b3988f2f38654eedac11fa475e56e92f [term 7 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 6 index: 34. Preceding OpId from leader: term: 7 index: 35. (index mismatch)
I20240502 14:01:21.351262 4886 consensus_queue.cc:1035] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240502 14:01:21.353844 4000 raft_consensus.cc:1261] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 7 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 6 index: 34. Preceding OpId from leader: term: 7 index: 35. (index mismatch)
I20240502 14:01:21.354249 4886 consensus_queue.cc:1035] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240502 14:01:21.490463 4150 raft_consensus.cc:1261] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [term 4 FOLLOWER]: Refusing update from remote peer b3988f2f38654eedac11fa475e56e92f: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240502 14:01:21.490823 4845 consensus_queue.cc:1035] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 14:01:21.493661 4078 raft_consensus.cc:1261] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 4 FOLLOWER]: Refusing update from remote peer b3988f2f38654eedac11fa475e56e92f: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240502 14:01:21.494127 4845 consensus_queue.cc:1035] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 14:01:21.535425 4478 raft_consensus.cc:1203] T fb4c3dcde8764e17bb084cb37fef706f P b3988f2f38654eedac11fa475e56e92f [term 2 LEARNER]: Deduplicated request from leader. Original: 2.31->[2.32-2.32] Dedup: 2.32->[]
I20240502 14:01:21.926009 4154 consensus_queue.cc:237] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 7.35, Last appended by leader: 34, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
I20240502 14:01:21.927122 4000 raft_consensus.cc:1261] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 7 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 7 index: 35. Preceding OpId from leader: term: 7 index: 36. (index mismatch)
I20240502 14:01:21.927122 4478 raft_consensus.cc:1261] T 144739b71bf6474d90399816120b3b59 P b3988f2f38654eedac11fa475e56e92f [term 7 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 7 index: 35. Preceding OpId from leader: term: 7 index: 36. (index mismatch)
I20240502 14:01:21.927532 4947 consensus_queue.cc:1035] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20240502 14:01:21.927773 4948 consensus_queue.cc:1035] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20240502 14:01:21.929283 4947 raft_consensus.cc:2932] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 7 LEADER]: Committing config change with OpId 7.36: config changed from index 34 to 36, VOTER 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66) evicted. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:21.929690 4000 raft_consensus.cc:2932] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 7 FOLLOWER]: Committing config change with OpId 7.36: config changed from index 34 to 36, VOTER 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66) evicted. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:21.929692 4478 raft_consensus.cc:2932] T 144739b71bf6474d90399816120b3b59 P b3988f2f38654eedac11fa475e56e92f [term 7 FOLLOWER]: Committing config change with OpId 7.36: config changed from index 34 to 36, VOTER 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66) evicted. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:21.930613 3890 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 144739b71bf6474d90399816120b3b59 with cas_config_opid_index 34: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:01:21.931401 3898 catalog_manager.cc:5484] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 reported cstate change: config changed from index 34 to 36, VOTER 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66) evicted. New cstate: current_term: 7 leader_uuid: "df655d53530449fc85de33de1c229815" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:01:21.933964 4067 tablet_service.cc:1514] Processing DeleteTablet for tablet 144739b71bf6474d90399816120b3b59 with delete_type TABLET_DATA_TOMBSTONED (TS 24fd5d6c84c1483ea8f39bb975e26e45 not found in new config with opid_index 36) from {username='slave'} at 127.0.0.1:59772
I20240502 14:01:21.934377 4952 tablet_replica.cc:330] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45: stopping tablet replica
I20240502 14:01:21.934501 4952 raft_consensus.cc:2229] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 7 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:21.934599 4952 raft_consensus.cc:2258] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 7 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:21.935518 4952 ts_tablet_manager.cc:1875] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:21.937427 4952 ts_tablet_manager.cc:1888] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 7.35
I20240502 14:01:21.937502 4952 log.cc:1196] T 144739b71bf6474d90399816120b3b59 P 24fd5d6c84c1483ea8f39bb975e26e45: Deleting WAL directory at /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-1-root/wals/144739b71bf6474d90399816120b3b59
I20240502 14:01:21.937924 3890 catalog_manager.cc:4830] TS 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251): tablet 144739b71bf6474d90399816120b3b59 (table test-workload [id=66735bdbe8fc4765af013422ce730ad0]) successfully deleted
I20240502 14:01:22.150612 4948 raft_consensus.cc:1050] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815: attempting to promote NON_VOTER b3988f2f38654eedac11fa475e56e92f to VOTER
I20240502 14:01:22.151078 4948 consensus_queue.cc:237] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 27, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
I20240502 14:01:22.152267 4478 raft_consensus.cc:1261] T fb4c3dcde8764e17bb084cb37fef706f P b3988f2f38654eedac11fa475e56e92f [term 2 LEARNER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240502 14:01:22.152294 4552 raft_consensus.cc:1261] T fb4c3dcde8764e17bb084cb37fef706f P e1de31532912433dbed75c2bb01bc42a [term 2 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240502 14:01:22.152377 4078 raft_consensus.cc:1261] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240502 14:01:22.152702 4939 consensus_queue.cc:1035] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 14:01:22.152913 4948 consensus_queue.cc:1035] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 14:01:22.153200 4947 consensus_queue.cc:1035] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 14:01:22.154675 4948 raft_consensus.cc:2932] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 32 to 33, b3988f2f38654eedac11fa475e56e92f (127.0.145.69) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:22.155022 4078 raft_consensus.cc:2932] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, b3988f2f38654eedac11fa475e56e92f (127.0.145.69) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:22.156201 4552 raft_consensus.cc:2932] T fb4c3dcde8764e17bb084cb37fef706f P e1de31532912433dbed75c2bb01bc42a [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, b3988f2f38654eedac11fa475e56e92f (127.0.145.69) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:22.156502 4478 raft_consensus.cc:2932] T fb4c3dcde8764e17bb084cb37fef706f P b3988f2f38654eedac11fa475e56e92f [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, b3988f2f38654eedac11fa475e56e92f (127.0.145.69) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:22.156752 3900 catalog_manager.cc:5484] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 reported cstate change: config changed from index 32 to 33, b3988f2f38654eedac11fa475e56e92f (127.0.145.69) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "df655d53530449fc85de33de1c229815" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:01:22.159935 4154 consensus_queue.cc:237] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
I20240502 14:01:22.160826 4552 raft_consensus.cc:1261] T fb4c3dcde8764e17bb084cb37fef706f P e1de31532912433dbed75c2bb01bc42a [term 2 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240502 14:01:22.160948 4478 raft_consensus.cc:1261] T fb4c3dcde8764e17bb084cb37fef706f P b3988f2f38654eedac11fa475e56e92f [term 2 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240502 14:01:22.161267 4939 consensus_queue.cc:1035] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 14:01:22.161427 4947 consensus_queue.cc:1035] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 14:01:22.162863 4947 raft_consensus.cc:2932] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 [term 2 LEADER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:22.163205 4552 raft_consensus.cc:2932] T fb4c3dcde8764e17bb084cb37fef706f P e1de31532912433dbed75c2bb01bc42a [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:22.165083 3890 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fb4c3dcde8764e17bb084cb37fef706f with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:01:22.165000 4478 raft_consensus.cc:2932] T fb4c3dcde8764e17bb084cb37fef706f P b3988f2f38654eedac11fa475e56e92f [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } }
I20240502 14:01:22.165138 3894 catalog_manager.cc:5484] T fb4c3dcde8764e17bb084cb37fef706f P df655d53530449fc85de33de1c229815 reported cstate change: config changed from index 33 to 34, VOTER 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66) evicted. New cstate: current_term: 2 leader_uuid: "df655d53530449fc85de33de1c229815" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:22.167520 4067 tablet_service.cc:1514] Processing DeleteTablet for tablet fb4c3dcde8764e17bb084cb37fef706f with delete_type TABLET_DATA_TOMBSTONED (TS 24fd5d6c84c1483ea8f39bb975e26e45 not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:59772
I20240502 14:01:22.167724 4952 tablet_replica.cc:330] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45: stopping tablet replica
I20240502 14:01:22.167824 4952 raft_consensus.cc:2229] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:22.167946 4952 raft_consensus.cc:2258] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:22.168418 4952 ts_tablet_manager.cc:1875] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:22.170037 4952 ts_tablet_manager.cc:1888] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.33
I20240502 14:01:22.170096 4952 log.cc:1196] T fb4c3dcde8764e17bb084cb37fef706f P 24fd5d6c84c1483ea8f39bb975e26e45: Deleting WAL directory at /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-1-root/wals/fb4c3dcde8764e17bb084cb37fef706f
I20240502 14:01:22.170452 3890 catalog_manager.cc:4830] TS 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251): tablet fb4c3dcde8764e17bb084cb37fef706f (table test-workload [id=66735bdbe8fc4765af013422ce730ad0]) successfully deleted
I20240502 14:01:22.691957 3957 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:01:22.693579 4150 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "144739b71bf6474d90399816120b3b59"
dest_uuid: "df655d53530449fc85de33de1c229815"
mode: GRACEFUL
new_leader_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
from {username='slave'} at 127.0.0.1:33630
I20240502 14:01:22.693732 4150 raft_consensus.cc:595] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 7 LEADER]: Received request to transfer leadership to TS bd060dd4339f4fa5a985596bf7dec0ae
I20240502 14:01:22.694463 4078 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "975becfb8df0430fa8c748a58dcbcfb1"
dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
mode: GRACEFUL
new_leader_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
from {username='slave'} at 127.0.0.1:59812
I20240502 14:01:22.694559 4078 raft_consensus.cc:595] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 LEADER]: Received request to transfer leadership to TS bd060dd4339f4fa5a985596bf7dec0ae
I20240502 14:01:22.694859 3957 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240502 14:01:22.694953 3957 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:01:22.801931 4939 raft_consensus.cc:979] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815: : Instructing follower bd060dd4339f4fa5a985596bf7dec0ae to start an election
I20240502 14:01:22.802028 4939 raft_consensus.cc:1067] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 7 LEADER]: Signalling peer bd060dd4339f4fa5a985596bf7dec0ae to start an election
I20240502 14:01:22.802429 4000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "144739b71bf6474d90399816120b3b59"
dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
from {username='slave'} at 127.0.0.1:57070
I20240502 14:01:22.802544 4000 raft_consensus.cc:482] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:22.802600 4000 raft_consensus.cc:3037] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 7 FOLLOWER]: Advancing to term 8
I20240502 14:01:22.803386 4000 raft_consensus.cc:504] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
I20240502 14:01:22.803712 4000 leader_election.cc:288] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 8 election: Requested vote from peers df655d53530449fc85de33de1c229815 (127.0.145.67:34309), b3988f2f38654eedac11fa475e56e92f (127.0.145.69:45021)
I20240502 14:01:22.803997 4478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144739b71bf6474d90399816120b3b59" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 8 candidate_status { last_received { term: 7 index: 36 } } ignore_live_leader: true dest_uuid: "b3988f2f38654eedac11fa475e56e92f"
I20240502 14:01:22.803988 4150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144739b71bf6474d90399816120b3b59" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 8 candidate_status { last_received { term: 7 index: 36 } } ignore_live_leader: true dest_uuid: "df655d53530449fc85de33de1c229815"
I20240502 14:01:22.804122 4478 raft_consensus.cc:3037] T 144739b71bf6474d90399816120b3b59 P b3988f2f38654eedac11fa475e56e92f [term 7 FOLLOWER]: Advancing to term 8
I20240502 14:01:22.804123 4150 raft_consensus.cc:3032] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 7 LEADER]: Stepping down as leader of term 7
I20240502 14:01:22.804186 4150 raft_consensus.cc:729] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: df655d53530449fc85de33de1c229815, State: Running, Role: LEADER
I20240502 14:01:22.804327 4150 consensus_queue.cc:260] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 7.36, Last appended by leader: 36, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
I20240502 14:01:22.804500 4150 raft_consensus.cc:3037] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 7 FOLLOWER]: Advancing to term 8
I20240502 14:01:22.805120 4478 raft_consensus.cc:2454] T 144739b71bf6474d90399816120b3b59 P b3988f2f38654eedac11fa475e56e92f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 8.
I20240502 14:01:22.805379 4150 raft_consensus.cc:2454] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 8.
I20240502 14:01:22.805408 3969 leader_election.cc:302] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3988f2f38654eedac11fa475e56e92f, bd060dd4339f4fa5a985596bf7dec0ae; no voters:
I20240502 14:01:22.805581 4919 raft_consensus.cc:2781] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 8 FOLLOWER]: Leader election won for term 8
I20240502 14:01:22.805739 4919 raft_consensus.cc:686] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [term 8 LEADER]: Becoming Leader. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Running, Role: LEADER
I20240502 14:01:22.805897 4919 consensus_queue.cc:237] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 7.36, Last appended by leader: 36, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
I20240502 14:01:22.807425 3898 catalog_manager.cc:5484] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae reported cstate change: term changed from 7 to 8, leader changed from df655d53530449fc85de33de1c229815 (127.0.145.67) to bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65). New cstate: current_term: 8 leader_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:23.168612 4478 consensus_queue.cc:237] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 4.34, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } }
I20240502 14:01:23.170096 4000 raft_consensus.cc:1261] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 4 FOLLOWER]: Refusing update from remote peer b3988f2f38654eedac11fa475e56e92f: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 4 index: 35. (index mismatch)
I20240502 14:01:23.170478 4401 raft_consensus.cc:1261] T b5172e93df3a4fb5a70188b0507639e6 P 5af426c57bc3434c9fbdcdff873a565c [term 4 FOLLOWER]: Refusing update from remote peer b3988f2f38654eedac11fa475e56e92f: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 4 index: 35. (index mismatch)
I20240502 14:01:23.170506 4966 consensus_queue.cc:1035] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240502 14:01:23.170845 4965 consensus_queue.cc:1035] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240502 14:01:23.172462 4954 raft_consensus.cc:2932] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [term 4 LEADER]: Committing config change with OpId 4.35: config changed from index 34 to 35, NON_VOTER e1de31532912433dbed75c2bb01bc42a (127.0.145.70) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } } }
I20240502 14:01:23.172832 4000 raft_consensus.cc:2932] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 4 FOLLOWER]: Committing config change with OpId 4.35: config changed from index 34 to 35, NON_VOTER e1de31532912433dbed75c2bb01bc42a (127.0.145.70) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } } }
I20240502 14:01:23.172865 4401 raft_consensus.cc:2932] T b5172e93df3a4fb5a70188b0507639e6 P 5af426c57bc3434c9fbdcdff873a565c [term 4 FOLLOWER]: Committing config change with OpId 4.35: config changed from index 34 to 35, NON_VOTER e1de31532912433dbed75c2bb01bc42a (127.0.145.70) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } } }
W20240502 14:01:23.174675 4440 consensus_peers.cc:485] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f -> Peer e1de31532912433dbed75c2bb01bc42a (127.0.145.70:41637): Couldn't send request to peer e1de31532912433dbed75c2bb01bc42a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5172e93df3a4fb5a70188b0507639e6. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:23.174957 3898 catalog_manager.cc:5484] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae reported cstate change: config changed from index 34 to 35, NON_VOTER e1de31532912433dbed75c2bb01bc42a (127.0.145.70) added. New cstate: current_term: 4 leader_uuid: "b3988f2f38654eedac11fa475e56e92f" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } } }
W20240502 14:01:23.175896 3952 auto_rebalancer.cc:253] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20240502 14:01:23.177450 3952 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 144739b71bf6474d90399816120b3b59, TS df655d53530449fc85de33de1c229815 -> TS 5af426c57bc3434c9fbdcdff873a565c move failed, destination replica disappeared from tablet's Raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
W20240502 14:01:23.177501 3952 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 144739b71bf6474d90399816120b3b59, TS df655d53530449fc85de33de1c229815 -> TS 5af426c57bc3434c9fbdcdff873a565c move failed, destination replica disappeared from tablet's Raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
W20240502 14:01:23.178938 3952 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 975becfb8df0430fa8c748a58dcbcfb1, TS bd060dd4339f4fa5a985596bf7dec0ae -> TS e1de31532912433dbed75c2bb01bc42a move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
W20240502 14:01:23.178985 3952 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 975becfb8df0430fa8c748a58dcbcfb1, TS bd060dd4339f4fa5a985596bf7dec0ae -> TS e1de31532912433dbed75c2bb01bc42a move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
W20240502 14:01:23.187574 3952 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet a4d0e9d6d8e7445a9367e0f3a31c8748, TS 24fd5d6c84c1483ea8f39bb975e26e45 -> TS 5af426c57bc3434c9fbdcdff873a565c move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } }
W20240502 14:01:23.187623 3952 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet a4d0e9d6d8e7445a9367e0f3a31c8748, TS 24fd5d6c84c1483ea8f39bb975e26e45 -> TS 5af426c57bc3434c9fbdcdff873a565c move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } }
W20240502 14:01:23.189061 3952 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 3d0d4aeaec264d04bae2c34e6edc6ec7, TS df655d53530449fc85de33de1c229815 -> TS b3988f2f38654eedac11fa475e56e92f move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } }
W20240502 14:01:23.189106 3952 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 3d0d4aeaec264d04bae2c34e6edc6ec7, TS df655d53530449fc85de33de1c229815 -> TS b3988f2f38654eedac11fa475e56e92f move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } }
I20240502 14:01:23.224532 4150 raft_consensus.cc:1261] T 144739b71bf6474d90399816120b3b59 P df655d53530449fc85de33de1c229815 [term 8 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 7 index: 36. Preceding OpId from leader: term: 8 index: 37. (index mismatch)
I20240502 14:01:23.224911 4919 consensus_queue.cc:1035] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20240502 14:01:23.227716 4478 raft_consensus.cc:1261] T 144739b71bf6474d90399816120b3b59 P b3988f2f38654eedac11fa475e56e92f [term 8 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 7 index: 36. Preceding OpId from leader: term: 8 index: 37. (index mismatch)
I20240502 14:01:23.228123 4960 consensus_queue.cc:1035] T 144739b71bf6474d90399816120b3b59 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20240502 14:01:23.270769 4946 raft_consensus.cc:979] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45: : Instructing follower bd060dd4339f4fa5a985596bf7dec0ae to start an election
I20240502 14:01:23.270874 4946 raft_consensus.cc:1067] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 LEADER]: Signalling peer bd060dd4339f4fa5a985596bf7dec0ae to start an election
I20240502 14:01:23.271220 4000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "975becfb8df0430fa8c748a58dcbcfb1"
dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
from {username='slave'} at 127.0.0.1:57072
I20240502 14:01:23.271324 4000 raft_consensus.cc:482] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:23.271381 4000 raft_consensus.cc:3037] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:01:23.272117 4000 raft_consensus.cc:504] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:23.272501 4000 leader_election.cc:288] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 3 election: Requested vote from peers 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:23.272809 4078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975becfb8df0430fa8c748a58dcbcfb1" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
I20240502 14:01:23.272825 4150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975becfb8df0430fa8c748a58dcbcfb1" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "df655d53530449fc85de33de1c229815"
I20240502 14:01:23.272923 4078 raft_consensus.cc:3032] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:01:23.272946 4150 raft_consensus.cc:3037] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:01:23.272969 4078 raft_consensus.cc:729] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 24fd5d6c84c1483ea8f39bb975e26e45, State: Running, Role: LEADER
I20240502 14:01:23.273079 4078 consensus_queue.cc:260] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:23.273191 4078 raft_consensus.cc:3037] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:01:23.273808 4150 raft_consensus.cc:2454] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 3.
I20240502 14:01:23.273980 4078 raft_consensus.cc:2454] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 3.
I20240502 14:01:23.274031 3972 leader_election.cc:302] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd060dd4339f4fa5a985596bf7dec0ae, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:23.274174 4960 raft_consensus.cc:2781] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:01:23.274256 4960 raft_consensus.cc:686] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 LEADER]: Becoming Leader. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Running, Role: LEADER
I20240502 14:01:23.274367 4960 consensus_queue.cc:237] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:23.275745 3903 catalog_manager.cc:5484] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae reported cstate change: term changed from 2 to 3, leader changed from 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66) to bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65). New cstate: current_term: 3 leader_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:23.770424 4979 ts_tablet_manager.cc:902] T b5172e93df3a4fb5a70188b0507639e6 P e1de31532912433dbed75c2bb01bc42a: Initiating tablet copy from peer b3988f2f38654eedac11fa475e56e92f (127.0.145.69:45021)
I20240502 14:01:23.771116 4979 tablet_copy_client.cc:308] T b5172e93df3a4fb5a70188b0507639e6 P e1de31532912433dbed75c2bb01bc42a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.69:45021
I20240502 14:01:23.774784 4488 tablet_copy_service.cc:140] P b3988f2f38654eedac11fa475e56e92f: Received BeginTabletCopySession request for tablet b5172e93df3a4fb5a70188b0507639e6 from peer e1de31532912433dbed75c2bb01bc42a ({username='slave'} at 127.0.0.1:44642)
I20240502 14:01:23.774880 4488 tablet_copy_service.cc:161] P b3988f2f38654eedac11fa475e56e92f: Beginning new tablet copy session on tablet b5172e93df3a4fb5a70188b0507639e6 from peer e1de31532912433dbed75c2bb01bc42a at {username='slave'} at 127.0.0.1:44642: session id = e1de31532912433dbed75c2bb01bc42a-b5172e93df3a4fb5a70188b0507639e6
I20240502 14:01:23.775765 4078 raft_consensus.cc:1261] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 14:01:23.775879 4488 tablet_copy_source_session.cc:205] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:23.776134 4919 consensus_queue.cc:1035] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:01:23.776556 4979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5172e93df3a4fb5a70188b0507639e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:23.778700 4979 tablet_copy_client.cc:783] T b5172e93df3a4fb5a70188b0507639e6 P e1de31532912433dbed75c2bb01bc42a: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:23.778825 4150 raft_consensus.cc:1261] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [term 3 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 14:01:23.778831 4979 tablet_copy_client.cc:647] T b5172e93df3a4fb5a70188b0507639e6 P e1de31532912433dbed75c2bb01bc42a: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:23.779198 4987 consensus_queue.cc:1035] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:01:23.780575 4983 tablet_copy_client.cc:1002] T b5172e93df3a4fb5a70188b0507639e6 P e1de31532912433dbed75c2bb01bc42a: tablet copy: Injecting latency into file download: 2000
I20240502 14:01:24.695173 3957 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:01:24.696942 4000 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "46b4a50dee854f62aac3a02889525b40"
dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
mode: GRACEFUL
new_leader_uuid: "e1de31532912433dbed75c2bb01bc42a"
from {username='slave'} at 127.0.0.1:57080
I20240502 14:01:24.697091 4000 raft_consensus.cc:595] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 LEADER]: Received request to transfer leadership to TS e1de31532912433dbed75c2bb01bc42a
I20240502 14:01:24.697749 4000 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "975becfb8df0430fa8c748a58dcbcfb1"
dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
mode: GRACEFUL
new_leader_uuid: "df655d53530449fc85de33de1c229815"
from {username='slave'} at 127.0.0.1:57080
I20240502 14:01:24.697853 4000 raft_consensus.cc:595] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 LEADER]: Received request to transfer leadership to TS df655d53530449fc85de33de1c229815
I20240502 14:01:24.698107 3957 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240502 14:01:24.698217 3957 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:01:24.755031 4919 raft_consensus.cc:979] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae: : Instructing follower df655d53530449fc85de33de1c229815 to start an election
I20240502 14:01:24.755136 4919 raft_consensus.cc:1067] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 LEADER]: Signalling peer df655d53530449fc85de33de1c229815 to start an election
I20240502 14:01:24.755647 4150 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "975becfb8df0430fa8c748a58dcbcfb1"
dest_uuid: "df655d53530449fc85de33de1c229815"
from {username='slave'} at 127.0.0.1:33594
I20240502 14:01:24.755781 4150 raft_consensus.cc:482] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:24.755837 4150 raft_consensus.cc:3037] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:01:24.756645 4150 raft_consensus.cc:504] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:24.757086 4150 leader_election.cc:288] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [CANDIDATE]: Term 4 election: Requested vote from peers bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667), 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251)
I20240502 14:01:24.757308 4000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975becfb8df0430fa8c748a58dcbcfb1" candidate_uuid: "df655d53530449fc85de33de1c229815" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
I20240502 14:01:24.757372 4078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975becfb8df0430fa8c748a58dcbcfb1" candidate_uuid: "df655d53530449fc85de33de1c229815" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
I20240502 14:01:24.757433 4000 raft_consensus.cc:3032] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:01:24.757473 4000 raft_consensus.cc:729] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Running, Role: LEADER
I20240502 14:01:24.757494 4078 raft_consensus.cc:3037] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:01:24.757602 4000 consensus_queue.cc:260] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:24.757764 4000 raft_consensus.cc:3037] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:01:24.758432 4078 raft_consensus.cc:2454] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df655d53530449fc85de33de1c229815 in term 4.
I20240502 14:01:24.758670 4123 leader_election.cc:302] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, df655d53530449fc85de33de1c229815; no voters:
I20240502 14:01:24.758687 4000 raft_consensus.cc:2454] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df655d53530449fc85de33de1c229815 in term 4.
I20240502 14:01:24.758847 4992 raft_consensus.cc:2781] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:01:24.758996 4992 raft_consensus.cc:686] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [term 4 LEADER]: Becoming Leader. State: Replica: df655d53530449fc85de33de1c229815, State: Running, Role: LEADER
I20240502 14:01:24.759135 4992 consensus_queue.cc:237] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:24.760545 3894 catalog_manager.cc:5484] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 reported cstate change: term changed from 3 to 4, leader changed from bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65) to df655d53530449fc85de33de1c229815 (127.0.145.67). New cstate: current_term: 4 leader_uuid: "df655d53530449fc85de33de1c229815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: HEALTHY } } }
I20240502 14:01:24.934803 4996 raft_consensus.cc:979] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae: : Instructing follower e1de31532912433dbed75c2bb01bc42a to start an election
I20240502 14:01:24.934916 4996 raft_consensus.cc:1067] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 LEADER]: Signalling peer e1de31532912433dbed75c2bb01bc42a to start an election
I20240502 14:01:24.935297 4552 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "46b4a50dee854f62aac3a02889525b40"
dest_uuid: "e1de31532912433dbed75c2bb01bc42a"
from {username='slave'} at 127.0.0.1:49836
I20240502 14:01:24.935421 4552 raft_consensus.cc:482] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:24.935493 4552 raft_consensus.cc:3037] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:01:24.936373 4552 raft_consensus.cc:504] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } }
I20240502 14:01:24.936709 4552 leader_election.cc:288] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a [CANDIDATE]: Term 3 election: Requested vote from peers bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667), 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251)
I20240502 14:01:24.936925 4000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46b4a50dee854f62aac3a02889525b40" candidate_uuid: "e1de31532912433dbed75c2bb01bc42a" candidate_term: 3 candidate_status { last_received { term: 2 index: 31 } } ignore_live_leader: true dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
I20240502 14:01:24.937041 4078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46b4a50dee854f62aac3a02889525b40" candidate_uuid: "e1de31532912433dbed75c2bb01bc42a" candidate_term: 3 candidate_status { last_received { term: 2 index: 31 } } ignore_live_leader: true dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
I20240502 14:01:24.937047 4000 raft_consensus.cc:3032] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:01:24.937145 4000 raft_consensus.cc:729] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Running, Role: LEADER
I20240502 14:01:24.937160 4078 raft_consensus.cc:3037] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:01:24.937276 4000 consensus_queue.cc:260] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } }
I20240502 14:01:24.937424 4000 raft_consensus.cc:3037] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:01:24.938043 4078 raft_consensus.cc:2454] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1de31532912433dbed75c2bb01bc42a in term 3.
I20240502 14:01:24.938215 4000 raft_consensus.cc:2454] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1de31532912433dbed75c2bb01bc42a in term 3.
I20240502 14:01:24.938318 4519 leader_election.cc:302] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, e1de31532912433dbed75c2bb01bc42a; no voters:
I20240502 14:01:24.938488 4918 raft_consensus.cc:2781] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:01:24.938673 4918 raft_consensus.cc:686] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a [term 3 LEADER]: Becoming Leader. State: Replica: e1de31532912433dbed75c2bb01bc42a, State: Running, Role: LEADER
I20240502 14:01:24.938804 4918 consensus_queue.cc:237] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } }
I20240502 14:01:24.940196 3894 catalog_manager.cc:5484] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a reported cstate change: term changed from 2 to 3, leader changed from bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65) to e1de31532912433dbed75c2bb01bc42a (127.0.145.70). New cstate: current_term: 3 leader_uuid: "e1de31532912433dbed75c2bb01bc42a" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:01:25.181990 4078 raft_consensus.cc:1261] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 4 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20240502 14:01:25.182340 4997 consensus_queue.cc:1035] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:01:25.185246 4000 raft_consensus.cc:1261] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 4 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20240502 14:01:25.185650 4997 consensus_queue.cc:1035] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:01:25.375182 4000 raft_consensus.cc:1261] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 FOLLOWER]: Refusing update from remote peer e1de31532912433dbed75c2bb01bc42a: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240502 14:01:25.375582 4918 consensus_queue.cc:1035] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 14:01:25.377758 4078 raft_consensus.cc:1261] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 FOLLOWER]: Refusing update from remote peer e1de31532912433dbed75c2bb01bc42a: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240502 14:01:25.378261 4918 consensus_queue.cc:1035] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 14:01:25.780902 4979 tablet_copy_client.cc:523] T b5172e93df3a4fb5a70188b0507639e6 P e1de31532912433dbed75c2bb01bc42a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:25.782423 4979 tablet_bootstrap.cc:492] T b5172e93df3a4fb5a70188b0507639e6 P e1de31532912433dbed75c2bb01bc42a: Bootstrap starting.
I20240502 14:01:25.803597 4979 tablet_bootstrap.cc:492] T b5172e93df3a4fb5a70188b0507639e6 P e1de31532912433dbed75c2bb01bc42a: Bootstrap replayed 1/1 log segments. Stats: ops{read=35 overwritten=0 applied=35 ignored=0} inserts{seen=83 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:25.803901 4979 tablet_bootstrap.cc:492] T b5172e93df3a4fb5a70188b0507639e6 P e1de31532912433dbed75c2bb01bc42a: Bootstrap complete.
I20240502 14:01:25.804080 4979 ts_tablet_manager.cc:1367] T b5172e93df3a4fb5a70188b0507639e6 P e1de31532912433dbed75c2bb01bc42a: Time spent bootstrapping tablet: real 0.022s user 0.020s sys 0.000s
I20240502 14:01:25.804563 4979 raft_consensus.cc:348] T b5172e93df3a4fb5a70188b0507639e6 P e1de31532912433dbed75c2bb01bc42a [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } }
I20240502 14:01:25.804694 4979 raft_consensus.cc:729] T b5172e93df3a4fb5a70188b0507639e6 P e1de31532912433dbed75c2bb01bc42a [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: e1de31532912433dbed75c2bb01bc42a, State: Initialized, Role: LEARNER
I20240502 14:01:25.804816 4979 consensus_queue.cc:260] T b5172e93df3a4fb5a70188b0507639e6 P e1de31532912433dbed75c2bb01bc42a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 35, Last appended: 4.35, Last appended by leader: 35, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } }
I20240502 14:01:25.805182 4979 ts_tablet_manager.cc:1398] T b5172e93df3a4fb5a70188b0507639e6 P e1de31532912433dbed75c2bb01bc42a: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 14:01:25.805639 4488 tablet_copy_service.cc:342] P b3988f2f38654eedac11fa475e56e92f: Request end of tablet copy session e1de31532912433dbed75c2bb01bc42a-b5172e93df3a4fb5a70188b0507639e6 received from {username='slave'} at 127.0.0.1:44642
I20240502 14:01:25.805716 4488 tablet_copy_service.cc:433] P b3988f2f38654eedac11fa475e56e92f: ending tablet copy session e1de31532912433dbed75c2bb01bc42a-b5172e93df3a4fb5a70188b0507639e6 on tablet b5172e93df3a4fb5a70188b0507639e6 with peer e1de31532912433dbed75c2bb01bc42a
I20240502 14:01:26.113544 4552 raft_consensus.cc:1203] T b5172e93df3a4fb5a70188b0507639e6 P e1de31532912433dbed75c2bb01bc42a [term 4 LEARNER]: Deduplicated request from leader. Original: 4.34->[4.35-4.35] Dedup: 4.35->[]
I20240502 14:01:26.634641 5019 raft_consensus.cc:1050] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f: attempting to promote NON_VOTER e1de31532912433dbed75c2bb01bc42a to VOTER
I20240502 14:01:26.635159 5019 consensus_queue.cc:237] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 4.35, Last appended by leader: 30, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } }
I20240502 14:01:26.636518 4401 raft_consensus.cc:1261] T b5172e93df3a4fb5a70188b0507639e6 P 5af426c57bc3434c9fbdcdff873a565c [term 4 FOLLOWER]: Refusing update from remote peer b3988f2f38654eedac11fa475e56e92f: Log matching property violated. Preceding OpId in replica: term: 4 index: 35. Preceding OpId from leader: term: 4 index: 36. (index mismatch)
I20240502 14:01:26.636534 4552 raft_consensus.cc:1261] T b5172e93df3a4fb5a70188b0507639e6 P e1de31532912433dbed75c2bb01bc42a [term 4 LEARNER]: Refusing update from remote peer b3988f2f38654eedac11fa475e56e92f: Log matching property violated. Preceding OpId in replica: term: 4 index: 35. Preceding OpId from leader: term: 4 index: 36. (index mismatch)
I20240502 14:01:26.636811 4000 raft_consensus.cc:1261] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 4 FOLLOWER]: Refusing update from remote peer b3988f2f38654eedac11fa475e56e92f: Log matching property violated. Preceding OpId in replica: term: 4 index: 35. Preceding OpId from leader: term: 4 index: 36. (index mismatch)
I20240502 14:01:26.636873 5019 consensus_queue.cc:1035] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20240502 14:01:26.637164 5021 consensus_queue.cc:1035] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20240502 14:01:26.637414 5019 consensus_queue.cc:1035] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20240502 14:01:26.639302 5022 raft_consensus.cc:2932] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [term 4 LEADER]: Committing config change with OpId 4.36: config changed from index 35 to 36, e1de31532912433dbed75c2bb01bc42a (127.0.145.70) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:26.639657 4552 raft_consensus.cc:2932] T b5172e93df3a4fb5a70188b0507639e6 P e1de31532912433dbed75c2bb01bc42a [term 4 FOLLOWER]: Committing config change with OpId 4.36: config changed from index 35 to 36, e1de31532912433dbed75c2bb01bc42a (127.0.145.70) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:26.639988 4000 raft_consensus.cc:2932] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 4 FOLLOWER]: Committing config change with OpId 4.36: config changed from index 35 to 36, e1de31532912433dbed75c2bb01bc42a (127.0.145.70) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:26.641659 4401 raft_consensus.cc:2932] T b5172e93df3a4fb5a70188b0507639e6 P 5af426c57bc3434c9fbdcdff873a565c [term 4 FOLLOWER]: Committing config change with OpId 4.36: config changed from index 35 to 36, e1de31532912433dbed75c2bb01bc42a (127.0.145.70) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:26.641975 3903 catalog_manager.cc:5484] T b5172e93df3a4fb5a70188b0507639e6 P e1de31532912433dbed75c2bb01bc42a reported cstate change: config changed from index 35 to 36, e1de31532912433dbed75c2bb01bc42a (127.0.145.70) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "b3988f2f38654eedac11fa475e56e92f" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:26.645350 4476 consensus_queue.cc:237] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 4.36, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } }
I20240502 14:01:26.646368 4552 raft_consensus.cc:1261] T b5172e93df3a4fb5a70188b0507639e6 P e1de31532912433dbed75c2bb01bc42a [term 4 FOLLOWER]: Refusing update from remote peer b3988f2f38654eedac11fa475e56e92f: Log matching property violated. Preceding OpId in replica: term: 4 index: 36. Preceding OpId from leader: term: 4 index: 37. (index mismatch)
I20240502 14:01:26.646643 4401 raft_consensus.cc:1261] T b5172e93df3a4fb5a70188b0507639e6 P 5af426c57bc3434c9fbdcdff873a565c [term 4 FOLLOWER]: Refusing update from remote peer b3988f2f38654eedac11fa475e56e92f: Log matching property violated. Preceding OpId in replica: term: 4 index: 36. Preceding OpId from leader: term: 4 index: 37. (index mismatch)
I20240502 14:01:26.646926 5019 consensus_queue.cc:1035] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20240502 14:01:26.647105 5022 consensus_queue.cc:1035] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20240502 14:01:26.648386 5021 raft_consensus.cc:2932] T b5172e93df3a4fb5a70188b0507639e6 P b3988f2f38654eedac11fa475e56e92f [term 4 LEADER]: Committing config change with OpId 4.37: config changed from index 36 to 37, VOTER bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65) evicted. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:26.648782 4552 raft_consensus.cc:2932] T b5172e93df3a4fb5a70188b0507639e6 P e1de31532912433dbed75c2bb01bc42a [term 4 FOLLOWER]: Committing config change with OpId 4.37: config changed from index 36 to 37, VOTER bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65) evicted. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:26.648864 4401 raft_consensus.cc:2932] T b5172e93df3a4fb5a70188b0507639e6 P 5af426c57bc3434c9fbdcdff873a565c [term 4 FOLLOWER]: Committing config change with OpId 4.37: config changed from index 36 to 37, VOTER bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65) evicted. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:26.650540 3900 catalog_manager.cc:5484] T b5172e93df3a4fb5a70188b0507639e6 P 5af426c57bc3434c9fbdcdff873a565c reported cstate change: config changed from index 36 to 37, VOTER bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65) evicted. New cstate: current_term: 4 leader_uuid: "b3988f2f38654eedac11fa475e56e92f" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } }
I20240502 14:01:26.650687 3946 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b5172e93df3a4fb5a70188b0507639e6 with cas_config_opid_index 36: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:01:26.652359 3989 tablet_service.cc:1514] Processing DeleteTablet for tablet b5172e93df3a4fb5a70188b0507639e6 with delete_type TABLET_DATA_TOMBSTONED (TS bd060dd4339f4fa5a985596bf7dec0ae not found in new config with opid_index 37) from {username='slave'} at 127.0.0.1:57056
I20240502 14:01:26.652704 5030 tablet_replica.cc:330] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae: stopping tablet replica
I20240502 14:01:26.652863 5030 raft_consensus.cc:2229] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:26.653029 5030 raft_consensus.cc:2258] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:26.653502 5030 ts_tablet_manager.cc:1875] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:26.655069 5030 ts_tablet_manager.cc:1888] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.36
I20240502 14:01:26.655130 5030 log.cc:1196] T b5172e93df3a4fb5a70188b0507639e6 P bd060dd4339f4fa5a985596bf7dec0ae: Deleting WAL directory at /tmp/dist-test-task7pYdx8/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714658376531600-581-0/minicluster-data/ts-0-root/wals/b5172e93df3a4fb5a70188b0507639e6
I20240502 14:01:26.655544 3889 catalog_manager.cc:4830] TS bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667): tablet b5172e93df3a4fb5a70188b0507639e6 (table test-workload [id=66735bdbe8fc4765af013422ce730ad0]) successfully deleted
I20240502 14:01:26.698453 3957 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:01:26.699899 4150 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "975becfb8df0430fa8c748a58dcbcfb1"
dest_uuid: "df655d53530449fc85de33de1c229815"
mode: GRACEFUL
new_leader_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
from {username='slave'} at 127.0.0.1:33630
I20240502 14:01:26.700022 4150 raft_consensus.cc:595] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [term 4 LEADER]: Received request to transfer leadership to TS bd060dd4339f4fa5a985596bf7dec0ae
I20240502 14:01:26.700356 3957 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240502 14:01:26.700482 3957 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:01:27.293462 5015 raft_consensus.cc:979] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815: : Instructing follower bd060dd4339f4fa5a985596bf7dec0ae to start an election
I20240502 14:01:27.293570 5015 raft_consensus.cc:1067] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [term 4 LEADER]: Signalling peer bd060dd4339f4fa5a985596bf7dec0ae to start an election
I20240502 14:01:27.293952 4000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "975becfb8df0430fa8c748a58dcbcfb1"
dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
from {username='slave'} at 127.0.0.1:57070
I20240502 14:01:27.294076 4000 raft_consensus.cc:482] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:27.294137 4000 raft_consensus.cc:3037] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:01:27.296384 4000 raft_consensus.cc:504] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:27.296732 4000 leader_election.cc:288] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 5 election: Requested vote from peers 24fd5d6c84c1483ea8f39bb975e26e45 (127.0.145.66:35251), df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
I20240502 14:01:27.297065 4078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975becfb8df0430fa8c748a58dcbcfb1" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 5 candidate_status { last_received { term: 4 index: 29 } } ignore_live_leader: true dest_uuid: "24fd5d6c84c1483ea8f39bb975e26e45"
I20240502 14:01:27.297063 4150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975becfb8df0430fa8c748a58dcbcfb1" candidate_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" candidate_term: 5 candidate_status { last_received { term: 4 index: 29 } } ignore_live_leader: true dest_uuid: "df655d53530449fc85de33de1c229815"
I20240502 14:01:27.297204 4150 raft_consensus.cc:3032] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 14:01:27.297204 4078 raft_consensus.cc:3037] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:01:27.297271 4150 raft_consensus.cc:729] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df655d53530449fc85de33de1c229815, State: Running, Role: LEADER
I20240502 14:01:27.297415 4150 consensus_queue.cc:260] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 29, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:27.297574 4150 raft_consensus.cc:3037] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:01:27.298175 4078 raft_consensus.cc:2454] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 5.
I20240502 14:01:27.298406 4150 raft_consensus.cc:2454] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd060dd4339f4fa5a985596bf7dec0ae in term 5.
I20240502 14:01:27.298471 3972 leader_election.cc:302] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fd5d6c84c1483ea8f39bb975e26e45, bd060dd4339f4fa5a985596bf7dec0ae; no voters:
I20240502 14:01:27.298660 4996 raft_consensus.cc:2781] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 5 FOLLOWER]: Leader election won for term 5
I20240502 14:01:27.298807 4996 raft_consensus.cc:686] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 5 LEADER]: Becoming Leader. State: Replica: bd060dd4339f4fa5a985596bf7dec0ae, State: Running, Role: LEADER
I20240502 14:01:27.298936 4996 consensus_queue.cc:237] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 29, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
I20240502 14:01:27.300321 3894 catalog_manager.cc:5484] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae reported cstate change: term changed from 4 to 5, leader changed from df655d53530449fc85de33de1c229815 (127.0.145.67) to bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65). New cstate: current_term: 5 leader_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: UNKNOWN } } }
W20240502 14:01:27.654172 3952 auto_rebalancer.cc:253] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20240502 14:01:27.655035 3952 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 975becfb8df0430fa8c748a58dcbcfb1, TS df655d53530449fc85de33de1c229815 -> TS 5af426c57bc3434c9fbdcdff873a565c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
W20240502 14:01:27.655090 3952 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 975becfb8df0430fa8c748a58dcbcfb1, TS df655d53530449fc85de33de1c229815 -> TS 5af426c57bc3434c9fbdcdff873a565c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } }
W20240502 14:01:27.655815 3952 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 144739b71bf6474d90399816120b3b59, TS df655d53530449fc85de33de1c229815 -> TS 5af426c57bc3434c9fbdcdff873a565c move failed, destination replica disappeared from tablet's Raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
W20240502 14:01:27.655865 3952 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 144739b71bf6474d90399816120b3b59, TS df655d53530449fc85de33de1c229815 -> TS 5af426c57bc3434c9fbdcdff873a565c move failed, destination replica disappeared from tablet's Raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
W20240502 14:01:27.656649 3952 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 3d0d4aeaec264d04bae2c34e6edc6ec7, TS 24fd5d6c84c1483ea8f39bb975e26e45 -> TS b3988f2f38654eedac11fa475e56e92f move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } }
W20240502 14:01:27.656689 3952 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 3d0d4aeaec264d04bae2c34e6edc6ec7, TS 24fd5d6c84c1483ea8f39bb975e26e45 -> TS b3988f2f38654eedac11fa475e56e92f move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: false } }
W20240502 14:01:27.657531 3952 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet c965f0e11bc1433f8ccdc20c98b4b4b9, TS bd060dd4339f4fa5a985596bf7dec0ae -> TS e1de31532912433dbed75c2bb01bc42a move failed, destination replica disappeared from tablet's Raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
W20240502 14:01:27.657585 3952 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet c965f0e11bc1433f8ccdc20c98b4b4b9, TS bd060dd4339f4fa5a985596bf7dec0ae -> TS e1de31532912433dbed75c2bb01bc42a move failed, destination replica disappeared from tablet's Raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } }
I20240502 14:01:27.739734 4150 raft_consensus.cc:1261] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [term 5 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 5 index: 30. (index mismatch)
I20240502 14:01:27.740084 4996 consensus_queue.cc:1035] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:01:27.742512 4078 raft_consensus.cc:1261] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 5 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 5 index: 30. (index mismatch)
I20240502 14:01:27.742908 4996 consensus_queue.cc:1035] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:01:28.660550 4478 consensus_queue.cc:237] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 31, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } }
I20240502 14:01:28.661944 4078 raft_consensus.cc:1261] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 4 FOLLOWER]: Refusing update from remote peer b3988f2f38654eedac11fa475e56e92f: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240502 14:01:28.662050 4150 raft_consensus.cc:1261] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [term 4 FOLLOWER]: Refusing update from remote peer b3988f2f38654eedac11fa475e56e92f: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
W20240502 14:01:28.662146 4443 consensus_peers.cc:485] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f -> Peer 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68:34601): Couldn't send request to peer 5af426c57bc3434c9fbdcdff873a565c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7908edf8bbe8419ab21d0c93cd2a3b35. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:28.662326 5019 consensus_queue.cc:1035] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 14:01:28.662516 5046 consensus_queue.cc:1035] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 14:01:28.664214 5047 raft_consensus.cc:2932] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f [term 4 LEADER]: Committing config change with OpId 4.33: config changed from index 30 to 33, NON_VOTER 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } } }
I20240502 14:01:28.664602 4078 raft_consensus.cc:2932] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 30 to 33, NON_VOTER 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } } }
I20240502 14:01:28.664585 4150 raft_consensus.cc:2932] T 7908edf8bbe8419ab21d0c93cd2a3b35 P df655d53530449fc85de33de1c229815 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 30 to 33, NON_VOTER 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } } }
I20240502 14:01:28.666705 3898 catalog_manager.cc:5484] T 7908edf8bbe8419ab21d0c93cd2a3b35 P b3988f2f38654eedac11fa475e56e92f reported cstate change: config changed from index 30 to 33, NON_VOTER 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) added. New cstate: current_term: 4 leader_uuid: "b3988f2f38654eedac11fa475e56e92f" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:28.667620 4552 consensus_queue.cc:237] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 31, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } }
I20240502 14:01:28.669274 4000 raft_consensus.cc:1261] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 FOLLOWER]: Refusing update from remote peer e1de31532912433dbed75c2bb01bc42a: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240502 14:01:28.670116 5054 consensus_queue.cc:1035] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 14:01:28.670275 4078 raft_consensus.cc:1261] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 FOLLOWER]: Refusing update from remote peer e1de31532912433dbed75c2bb01bc42a: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240502 14:01:28.670624 5055 consensus_queue.cc:1035] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 14:01:28.671857 5054 raft_consensus.cc:2932] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a [term 3 LEADER]: Committing config change with OpId 3.33: config changed from index 31 to 33, NON_VOTER 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } } }
I20240502 14:01:28.672317 4000 raft_consensus.cc:2932] T 46b4a50dee854f62aac3a02889525b40 P bd060dd4339f4fa5a985596bf7dec0ae [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 31 to 33, NON_VOTER 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } } }
W20240502 14:01:28.673789 4519 consensus_peers.cc:485] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a -> Peer 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68:34601): Couldn't send request to peer 5af426c57bc3434c9fbdcdff873a565c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 46b4a50dee854f62aac3a02889525b40. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:28.674012 4078 raft_consensus.cc:2932] T 46b4a50dee854f62aac3a02889525b40 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 31 to 33, NON_VOTER 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } } }
I20240502 14:01:28.674021 3894 catalog_manager.cc:5484] T 46b4a50dee854f62aac3a02889525b40 P e1de31532912433dbed75c2bb01bc42a reported cstate change: config changed from index 31 to 33, NON_VOTER 5af426c57bc3434c9fbdcdff873a565c (127.0.145.68) added. New cstate: current_term: 3 leader_uuid: "e1de31532912433dbed75c2bb01bc42a" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5af426c57bc3434c9fbdcdff873a565c" member_type: NON_VOTER last_known_addr { host: "127.0.145.68" port: 34601 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:28.674906 4000 consensus_queue.cc:237] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 5.30, Last appended by leader: 29, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } }
I20240502 14:01:28.676120 4150 raft_consensus.cc:1261] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [term 5 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 5 index: 30. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
W20240502 14:01:28.676126 3969 consensus_peers.cc:485] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae -> Peer b3988f2f38654eedac11fa475e56e92f (127.0.145.69:45021): Couldn't send request to peer b3988f2f38654eedac11fa475e56e92f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 975becfb8df0430fa8c748a58dcbcfb1. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:28.676163 4078 raft_consensus.cc:1261] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 5 FOLLOWER]: Refusing update from remote peer bd060dd4339f4fa5a985596bf7dec0ae: Log matching property violated. Preceding OpId in replica: term: 5 index: 30. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
I20240502 14:01:28.676544 5044 consensus_queue.cc:1035] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:01:28.676870 4996 consensus_queue.cc:1035] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:01:28.677960 5043 raft_consensus.cc:2932] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 5 LEADER]: Committing config change with OpId 5.31: config changed from index -1 to 31, NON_VOTER b3988f2f38654eedac11fa475e56e92f (127.0.145.69) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } } }
I20240502 14:01:28.678546 4150 raft_consensus.cc:2932] T 975becfb8df0430fa8c748a58dcbcfb1 P df655d53530449fc85de33de1c229815 [term 5 FOLLOWER]: Committing config change with OpId 5.31: config changed from index -1 to 31, NON_VOTER b3988f2f38654eedac11fa475e56e92f (127.0.145.69) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } } }
I20240502 14:01:28.679826 4154 consensus_queue.cc:237] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 31, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } }
I20240502 14:01:28.679813 3898 catalog_manager.cc:5484] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae reported cstate change: config changed from index -1 to 31, NON_VOTER b3988f2f38654eedac11fa475e56e92f (127.0.145.69) added. New cstate: current_term: 5 leader_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:28.679317 4078 raft_consensus.cc:2932] T 975becfb8df0430fa8c748a58dcbcfb1 P 24fd5d6c84c1483ea8f39bb975e26e45 [term 5 FOLLOWER]: Committing config change with OpId 5.31: config changed from index -1 to 31, NON_VOTER b3988f2f38654eedac11fa475e56e92f (127.0.145.69) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } } peers { permanent_uuid: "24fd5d6c84c1483ea8f39bb975e26e45" member_type: VOTER last_known_addr { host: "127.0.145.66" port: 35251 } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } attrs { replace: true } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: NON_VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: true } } }
I20240502 14:01:28.681201 4478 raft_consensus.cc:1261] T c965f0e11bc1433f8ccdc20c98b4b4b9 P b3988f2f38654eedac11fa475e56e92f [term 5 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 5 index: 34. Preceding OpId from leader: term: 5 index: 35. (index mismatch)
I20240502 14:01:28.681339 4000 raft_consensus.cc:1261] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 5 FOLLOWER]: Refusing update from remote peer df655d53530449fc85de33de1c229815: Log matching property violated. Preceding OpId in replica: term: 5 index: 34. Preceding OpId from leader: term: 5 index: 35. (index mismatch)
W20240502 14:01:28.681350 4120 consensus_peers.cc:485] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 -> Peer e1de31532912433dbed75c2bb01bc42a (127.0.145.70:41637): Couldn't send request to peer e1de31532912433dbed75c2bb01bc42a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c965f0e11bc1433f8ccdc20c98b4b4b9. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:28.681666 5063 consensus_queue.cc:1035] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240502 14:01:28.681838 5015 consensus_queue.cc:1035] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240502 14:01:28.683418 5015 raft_consensus.cc:2932] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 [term 5 LEADER]: Committing config change with OpId 5.35: config changed from index 34 to 35, NON_VOTER e1de31532912433dbed75c2bb01bc42a (127.0.145.70) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } } }
I20240502 14:01:28.684070 4478 raft_consensus.cc:2932] T c965f0e11bc1433f8ccdc20c98b4b4b9 P b3988f2f38654eedac11fa475e56e92f [term 5 FOLLOWER]: Committing config change with OpId 5.35: config changed from index 34 to 35, NON_VOTER e1de31532912433dbed75c2bb01bc42a (127.0.145.70) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } } }
I20240502 14:01:28.685066 4000 raft_consensus.cc:2932] T c965f0e11bc1433f8ccdc20c98b4b4b9 P bd060dd4339f4fa5a985596bf7dec0ae [term 5 FOLLOWER]: Committing config change with OpId 5.35: config changed from index 34 to 35, NON_VOTER e1de31532912433dbed75c2bb01bc42a (127.0.145.70) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } } }
I20240502 14:01:28.685720 3900 catalog_manager.cc:5484] T c965f0e11bc1433f8ccdc20c98b4b4b9 P df655d53530449fc85de33de1c229815 reported cstate change: config changed from index 34 to 35, NON_VOTER e1de31532912433dbed75c2bb01bc42a (127.0.145.70) added. New cstate: current_term: 5 leader_uuid: "df655d53530449fc85de33de1c229815" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "bd060dd4339f4fa5a985596bf7dec0ae" member_type: VOTER last_known_addr { host: "127.0.145.65" port: 46667 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df655d53530449fc85de33de1c229815" member_type: VOTER last_known_addr { host: "127.0.145.67" port: 34309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3988f2f38654eedac11fa475e56e92f" member_type: VOTER last_known_addr { host: "127.0.145.69" port: 45021 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1de31532912433dbed75c2bb01bc42a" member_type: NON_VOTER last_known_addr { host: "127.0.145.70" port: 41637 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:01:28.700723 3957 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:01:28.702553 4000 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "975becfb8df0430fa8c748a58dcbcfb1"
dest_uuid: "bd060dd4339f4fa5a985596bf7dec0ae"
mode: GRACEFUL
new_leader_uuid: "df655d53530449fc85de33de1c229815"
from {username='slave'} at 127.0.0.1:57080
I20240502 14:01:28.702651 4000 raft_consensus.cc:595] T 975becfb8df0430fa8c748a58dcbcfb1 P bd060dd4339f4fa5a985596bf7dec0ae [term 5 LEADER]: Received request to transfer leadership to TS df655d53530449fc85de33de1c229815
I20240502 14:01:28.702905 3957 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240502 14:01:28.703004 3957 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:01:29.062048 5067 ts_tablet_manager.cc:902] T 975becfb8df0430fa8c748a58dcbcfb1 P b3988f2f38654eedac11fa475e56e92f: Initiating tablet copy from peer bd060dd4339f4fa5a985596bf7dec0ae (127.0.145.65:46667)
F20240502 14:01:29.062278 5067 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-144739b71bf6474d90399816120b3b59.queue_time_us, tablet-download-pool-144739b71bf6474d90399816120b3b59.queue_time_us) (tablet-download-pool-144739b71bf6474d90399816120b3b59.run_cpu_time_us, tablet-download-pool-144739b71bf6474d90399816120b3b59.run_cpu_time_us) (tablet-download-pool-144739b71bf6474d90399816120b3b59.run_wall_time_us, tablet-download-pool-144739b71bf6474d90399816120b3b59.run_wall_time_us) (tablet-download-pool-3d0d4aeaec264d04bae2c34e6edc6ec7.queue_time_us, tablet-download-pool-3d0d4aeaec264d04bae2c34e6edc6ec7.queue_time_us) (tablet-download-pool-3d0d4aeaec264d04bae2c34e6edc6ec7.run_cpu_time_us, tablet-download-pool-3d0d4aeaec264d04bae2c34e6edc6ec7.run_cpu_time_us) (tablet-download-pool-3d0d4aeaec264d04bae2c34e6edc6ec7.run_wall_time_us, tablet-download-pool-3d0d4aeaec264d04bae2c34e6edc6ec7.run_wall_time_us) (tablet-download-pool-46b4a50dee854f62aac3a02889525b40.queue_time_us, tablet-download-pool-46b4a50dee854f62aac3a02889525b40.queue_time_us) (tablet-download-pool-46b4a50dee854f62aac3a02889525b40.run_cpu_time_us, tablet-download-pool-46b4a50dee854f62aac3a02889525b40.run_cpu_time_us) (tablet-download-pool-46b4a50dee854f62aac3a02889525b40.run_wall_time_us, tablet-download-pool-46b4a50dee854f62aac3a02889525b40.run_wall_time_us) (tablet-download-pool-4c8e99e6b3a2407296d6fedfe6516947.queue_time_us, tablet-download-pool-4c8e99e6b3a2407296d6fedfe6516947.queue_time_us) (tablet-download-pool-4c8e99e6b3a2407296d6fedfe6516947.run_cpu_time_us, tablet-download-pool-4c8e99e6b3a2407296d6fedfe6516947.run_cpu_time_us) (tablet-download-pool-4c8e99e6b3a2407296d6fedfe6516947.run_wall_time_us, tablet-download-pool-4c8e99e6b3a2407296d6fedfe6516947.run_wall_time_us) (tablet-download-pool-7908edf8bbe8419ab21d0c93cd2a3b35.queue_time_us, tablet-download-pool-7908edf8bbe8419ab21d0c93cd2a3b35.queue_time_us) (tablet-download-pool-7908edf8bbe8419ab21d0c93cd2a3b35.run_cpu_time_us, tablet-download-pool-7908edf8bbe8419ab21d0c93cd2a3b35.run_cpu_time_us) (tablet-download-pool-7908edf8bbe8419ab21d0c93cd2a3b35.run_wall_time_us, tablet-download-pool-7908edf8bbe8419ab21d0c93cd2a3b35.run_wall_time_us) (tablet-download-pool-7ac79a5b1f514bdcb36e0db4ff6e3a59.queue_time_us, tablet-download-pool-7ac79a5b1f514bdcb36e0db4ff6e3a59.queue_time_us) (tablet-download-pool-7ac79a5b1f514bdcb36e0db4ff6e3a59.run_cpu_time_us, tablet-download-pool-7ac79a5b1f514bdcb36e0db4ff6e3a59.run_cpu_time_us) (tablet-download-pool-7ac79a5b1f514bdcb36e0db4ff6e3a59.run_wall_time_us, tablet-download-pool-7ac79a5b1f514bdcb36e0db4ff6e3a59.run_wall_time_us) (tablet-download-pool-975becfb8df0430fa8c748a58dcbcfb1.queue_time_us, tablet-download-pool-975becfb8df0430fa8c748a58dcbcfb1.queue_time_us) (tablet-download-pool-a4d0e9d6d8e7445a9367e0f3a31c8748.queue_time_us, tablet-download-pool-a4d0e9d6d8e7445a9367e0f3a31c8748.queue_time_us) (tablet-download-pool-a4d0e9d6d8e7445a9367e0f3a31c8748.run_cpu_time_us, tablet-download-pool-a4d0e9d6d8e7445a9367e0f3a31c8748.run_cpu_time_us) (tablet-download-pool-a4d0e9d6d8e7445a9367e0f3a31c8748.run_wall_time_us, tablet-download-pool-a4d0e9d6d8e7445a9367e0f3a31c8748.run_wall_time_us) (tablet-download-pool-b5172e93df3a4fb5a70188b0507639e6.queue_time_us, tablet-download-pool-b5172e93df3a4fb5a70188b0507639e6.queue_time_us) (tablet-download-pool-b5172e93df3a4fb5a70188b0507639e6.run_cpu_time_us, tablet-download-pool-b5172e93df3a4fb5a70188b0507639e6.run_cpu_time_us) (tablet-download-pool-b5172e93df3a4fb5a70188b0507639e6.run_wall_time_us, tablet-download-pool-b5172e93df3a4fb5a70188b0507639e6.run_wall_time_us) (tablet-download-pool-c965f0e11bc1433f8ccdc20c98b4b4b9.queue_time_us, tablet-download-pool-c965f0e11bc1433f8ccdc20c98b4b4b9.queue_time_us) (tablet-download-pool-c965f0e11bc1433f8ccdc20c98b4b4b9.run_cpu_time_us, tablet-download-pool-c965f0e11bc1433f8ccdc20c98b4b4b9.run_cpu_time_us) (tablet-download-pool-c965f0e11bc1433f8ccdc20c98b4b4b9.run_wall_time_us, tablet-download-pool-c965f0e11bc1433f8ccdc20c98b4b4b9.run_wall_time_us) (tablet-download-pool-dba8942501c440b18f71c7106878d0c4.queue_time_us, tablet-download-pool-dba8942501c440b18f71c7106878d0c4.queue_time_us) (tablet-download-pool-dba8942501c440b18f71c7106878d0c4.run_cpu_time_us, tablet-download-pool-dba8942501c440b18f71c7106878d0c4.run_cpu_time_us) (tablet-download-pool-dba8942501c440b18f71c7106878d0c4.run_wall_time_us, tablet-download-pool-dba8942501c440b18f71c7106878d0c4.run_wall_time_us) (tablet-download-pool-e6ca4998e3634ba789210e0c072e29b0.queue_time_us, tablet-download-pool-e6ca4998e3634ba789210e0c072e29b0.queue_time_us) (tablet-download-pool-e6ca4998e3634ba789210e0c072e29b0.run_cpu_time_us, tablet-download-pool-e6ca4998e3634ba789210e0c072e29b0.run_cpu_time_us) (tablet-download-pool-e6ca4998e3634ba789210e0c072e29b0.run_wall_time_us, tablet-download-pool-e6ca4998e3634ba789210e0c072e29b0.run_wall_time_us) (tablet-download-pool-fb4c3dcde8764e17bb084cb37fef706f.queue_time_us, tablet-download-pool-fb4c3dcde8764e17bb084cb37fef706f.queue_time_us) (tablet-download-pool-fb4c3dcde8764e17bb084cb37fef706f.run_cpu_time_us, tablet-download-pool-fb4c3dcde8764e17bb084cb37fef706f.run_cpu_time_us) (tablet-download-pool-fb4c3dcde8764e17bb084cb37fef706f.run_wall_time_us, tablet-download-pool-fb4c3dcde8764e17bb084cb37fef706f.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
*** Aborted at 1714658489 (unix time) try "date -d @1714658489" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800000245) received by PID 581 (TID 0x7f023aa16700) from PID 581; stack trace: ***
@ 0x7f02ec4bf040 (unknown) at ??:0
@ 0x7f02ec4befb7 gsignal at ??:0
@ 0x7f02ec4c0921 abort at ??:0
@ 0x7f02ed1c3dcd google::LogMessage::Fail() at ??:0
@ 0x7f02ed1c7b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f02ed1c37cc google::LogMessage::Flush() at ??:0
@ 0x7f02ed1c4f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f02edd1b044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f02edcfff98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f02edcfdefc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f02eec7a77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f02eec83a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f02eed0476f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f02eed184f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240502 14:01:29.117265 5082 ts_tablet_manager.cc:902] T c965f0e11bc1433f8ccdc20c98b4b4b9 P e1de31532912433dbed75c2bb01bc42a: Initiating tablet copy from peer df655d53530449fc85de33de1c229815 (127.0.145.67:34309)
@ 0x7f02eed03226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240502 14:01:29.122731 5084 ts_tablet_manager.cc:902] T 7908edf8bbe8419ab21d0c93cd2a3b35 P 5af426c57bc3434c9fbdcdff873a565c: Initiating tablet copy from peer b3988f2f38654eedac11fa475e56e92f (127.0.145.69:45021)
@ 0x7f02eed12c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f02ef26d3cc std::function<>::operator()() at ??:0
@ 0x7f02edd02e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f02edd036ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f02edd0503b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f02ef26d3cc std::function<>::operator()() at ??:0
I20240502 14:01:29.146739 5090 ts_tablet_manager.cc:902] T 46b4a50dee854f62aac3a02889525b40 P 5af426c57bc3434c9fbdcdff873a565c: Initiating tablet copy from peer e1de31532912433dbed75c2bb01bc42a (127.0.145.70:41637)
@ 0x7f02edcf2807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f02ea4f46db start_thread at ??:0
@ 0x7f02ec5a171f clone at ??:0