[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 14:08:49.869813 16439 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.13.254:46433,127.16.13.253:40047,127.16.13.252:35075
I20240501 14:08:49.870774 16439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:49.871289 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:49.878822 16445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.878986 16444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:49.879853 16439 server_base.cc:1034] running on GCE node
W20240501 14:08:49.880419 16447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:49.880965 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:08:49.881069 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:08:49.881103 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572529881101 us; error 0 us; skew 500 ppm
I20240501 14:08:49.882974 16439 webserver.cc:460] Webserver started at http://127.16.13.254:40127/ using document root <none> and password file <none>
I20240501 14:08:49.883486 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:49.883539 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:49.883716 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:49.885440 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714572529860089-16439-0/minicluster-data/master-0-root/instance:
uuid: "0d83058ad11c489aba83f7bb5a48d430"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-mgxc"
I20240501 14:08:49.888862 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:08:49.890913 16452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.892058 16439 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:08:49.892170 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714572529860089-16439-0/minicluster-data/master-0-root
uuid: "0d83058ad11c489aba83f7bb5a48d430"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-mgxc"
I20240501 14:08:49.892278 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714572529860089-16439-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714572529860089-16439-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714572529860089-16439-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:49.907213 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:49.907816 16439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:49.907981 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:49.914820 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.254:46433
I20240501 14:08:49.914824 16504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.254:46433 every 8 connection(s)
I20240501 14:08:49.916006 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:49.917865 16507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.918588 16508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:49.918601 16505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 14:08:49.919301 16510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:49.919953 16439 server_base.cc:1034] running on GCE node
I20240501 14:08:49.920140 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:08:49.920185 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:08:49.920202 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572529920201 us; error 0 us; skew 500 ppm
I20240501 14:08:49.921223 16439 webserver.cc:460] Webserver started at http://127.16.13.253:35373/ using document root <none> and password file <none>
I20240501 14:08:49.921360 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:49.921402 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:49.921483 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:49.921833 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714572529860089-16439-0/minicluster-data/master-1-root/instance:
uuid: "eb4af6c1f076452593784c583c9f2c50"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-mgxc"
I20240501 14:08:49.923579 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240501 14:08:49.923538 16505 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.13.254" port: 46433 } has no permanent_uuid. Determining permanent_uuid...
I20240501 14:08:49.924561 16515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.924798 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240501 14:08:49.924861 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714572529860089-16439-0/minicluster-data/master-1-root
uuid: "eb4af6c1f076452593784c583c9f2c50"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-mgxc"
I20240501 14:08:49.924926 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714572529860089-16439-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714572529860089-16439-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714572529860089-16439-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:49.931742 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:49.932204 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:49.934680 16505 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.13.253" port: 40047 } has no permanent_uuid. Determining permanent_uuid...
W20240501 14:08:49.935544 16454 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.13.253:40047: connect: Connection refused (error 111)
W20240501 14:08:49.936417 16505 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.16.13.253:40047: Network error: Client connection negotiation failed: client connection to 127.16.13.253:40047: connect: Connection refused (error 111)
I20240501 14:08:49.937673 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.253:40047
I20240501 14:08:49.937747 16570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.253:40047 every 8 connection(s)
I20240501 14:08:49.938313 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 14:08:49.938526 16571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:49.939755 16571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.13.254" port: 46433 } has no permanent_uuid. Determining permanent_uuid...
W20240501 14:08:49.940176 16573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.941211 16576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.942005 16574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:49.942577 16439 server_base.cc:1034] running on GCE node
I20240501 14:08:49.942775 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:08:49.942826 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:08:49.942849 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572529942848 us; error 0 us; skew 500 ppm
I20240501 14:08:49.943864 16439 webserver.cc:460] Webserver started at http://127.16.13.252:39631/ using document root <none> and password file <none>
I20240501 14:08:49.944056 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:49.944119 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:49.944182 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:49.944391 16571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.13.253" port: 40047 } has no permanent_uuid. Determining permanent_uuid...
I20240501 14:08:49.944630 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714572529860089-16439-0/minicluster-data/master-2-root/instance:
uuid: "6c61b31cfd744e88a5fedcdd8838a81d"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-mgxc"
I20240501 14:08:49.946316 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 14:08:49.947584 16583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.947862 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 14:08:49.947960 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714572529860089-16439-0/minicluster-data/master-2-root
uuid: "6c61b31cfd744e88a5fedcdd8838a81d"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-mgxc"
I20240501 14:08:49.948030 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714572529860089-16439-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714572529860089-16439-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714572529860089-16439-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:49.948619 16571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.13.252" port: 35075 } has no permanent_uuid. Determining permanent_uuid...
W20240501 14:08:49.949849 16571 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.16.13.252:35075: Network error: Client connection negotiation failed: client connection to 127.16.13.252:35075: connect: Connection refused (error 111)
I20240501 14:08:49.957953 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:49.958323 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:49.962917 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.252:35075
I20240501 14:08:49.962978 16636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.252:35075 every 8 connection(s)
I20240501 14:08:49.963366 16439 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 14:08:49.963791 16637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:49.964869 16637 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.13.254" port: 46433 } has no permanent_uuid. Determining permanent_uuid...
I20240501 14:08:49.969060 16637 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.13.253" port: 40047 } has no permanent_uuid. Determining permanent_uuid...
I20240501 14:08:49.972688 16637 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.13.252" port: 35075 } has no permanent_uuid. Determining permanent_uuid...
I20240501 14:08:49.978313 16637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d: Bootstrap starting.
I20240501 14:08:49.980646 16637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:49.981428 16637 log.cc:824] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:49.982825 16637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d: No bootstrap required, opened a new log
I20240501 14:08:49.985217 16637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d83058ad11c489aba83f7bb5a48d430" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 46433 } } peers { permanent_uuid: "eb4af6c1f076452593784c583c9f2c50" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 40047 } } peers { permanent_uuid: "6c61b31cfd744e88a5fedcdd8838a81d" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 35075 } }
I20240501 14:08:49.985357 16637 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:49.985409 16637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c61b31cfd744e88a5fedcdd8838a81d, State: Initialized, Role: FOLLOWER
I20240501 14:08:49.985589 16505 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.13.253" port: 40047 } attempt: 1
I20240501 14:08:49.985833 16637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d83058ad11c489aba83f7bb5a48d430" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 46433 } } peers { permanent_uuid: "eb4af6c1f076452593784c583c9f2c50" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 40047 } } peers { permanent_uuid: "6c61b31cfd744e88a5fedcdd8838a81d" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 35075 } }
I20240501 14:08:49.986305 16642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d83058ad11c489aba83f7bb5a48d430" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 46433 } } peers { permanent_uuid: "eb4af6c1f076452593784c583c9f2c50" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 40047 } } peers { permanent_uuid: "6c61b31cfd744e88a5fedcdd8838a81d" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 35075 } } }
I20240501 14:08:49.986444 16642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:08:49.986953 16637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:08:49.989828 16505 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.13.252" port: 35075 } has no permanent_uuid. Determining permanent_uuid...
W20240501 14:08:49.991951 16653 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 14:08:49.992480 16653 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 14:08:49.994455 16505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d83058ad11c489aba83f7bb5a48d430: Bootstrap starting.
I20240501 14:08:49.995492 16505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d83058ad11c489aba83f7bb5a48d430: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:49.996762 16505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d83058ad11c489aba83f7bb5a48d430: No bootstrap required, opened a new log
I20240501 14:08:49.997218 16505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d83058ad11c489aba83f7bb5a48d430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d83058ad11c489aba83f7bb5a48d430" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 46433 } } peers { permanent_uuid: "eb4af6c1f076452593784c583c9f2c50" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 40047 } } peers { permanent_uuid: "6c61b31cfd744e88a5fedcdd8838a81d" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 35075 } }
I20240501 14:08:49.997331 16505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d83058ad11c489aba83f7bb5a48d430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:49.997372 16505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d83058ad11c489aba83f7bb5a48d430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d83058ad11c489aba83f7bb5a48d430, State: Initialized, Role: FOLLOWER
I20240501 14:08:49.997498 16505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d83058ad11c489aba83f7bb5a48d430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d83058ad11c489aba83f7bb5a48d430" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 46433 } } peers { permanent_uuid: "eb4af6c1f076452593784c583c9f2c50" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 40047 } } peers { permanent_uuid: "6c61b31cfd744e88a5fedcdd8838a81d" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 35075 } }
I20240501 14:08:49.997803 16656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d83058ad11c489aba83f7bb5a48d430 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d83058ad11c489aba83f7bb5a48d430" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 46433 } } peers { permanent_uuid: "eb4af6c1f076452593784c583c9f2c50" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 40047 } } peers { permanent_uuid: "6c61b31cfd744e88a5fedcdd8838a81d" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 35075 } } }
I20240501 14:08:49.997895 16656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d83058ad11c489aba83f7bb5a48d430 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:08:49.997990 16505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d83058ad11c489aba83f7bb5a48d430 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:08:49.999950 16439 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20240501 14:08:50.000586 16667 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0d83058ad11c489aba83f7bb5a48d430: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 14:08:50.000648 16667 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 14:08:50.002012 16571 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.13.252" port: 35075 } attempt: 1
I20240501 14:08:50.007227 16571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb4af6c1f076452593784c583c9f2c50: Bootstrap starting.
I20240501 14:08:50.008280 16571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb4af6c1f076452593784c583c9f2c50: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:50.009457 16571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb4af6c1f076452593784c583c9f2c50: No bootstrap required, opened a new log
I20240501 14:08:50.009886 16571 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb4af6c1f076452593784c583c9f2c50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d83058ad11c489aba83f7bb5a48d430" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 46433 } } peers { permanent_uuid: "eb4af6c1f076452593784c583c9f2c50" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 40047 } } peers { permanent_uuid: "6c61b31cfd744e88a5fedcdd8838a81d" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 35075 } }
I20240501 14:08:50.009992 16571 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb4af6c1f076452593784c583c9f2c50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:50.010033 16571 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb4af6c1f076452593784c583c9f2c50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb4af6c1f076452593784c583c9f2c50, State: Initialized, Role: FOLLOWER
I20240501 14:08:50.010159 16571 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb4af6c1f076452593784c583c9f2c50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d83058ad11c489aba83f7bb5a48d430" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 46433 } } peers { permanent_uuid: "eb4af6c1f076452593784c583c9f2c50" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 40047 } } peers { permanent_uuid: "6c61b31cfd744e88a5fedcdd8838a81d" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 35075 } }
I20240501 14:08:50.010486 16669 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb4af6c1f076452593784c583c9f2c50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d83058ad11c489aba83f7bb5a48d430" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 46433 } } peers { permanent_uuid: "eb4af6c1f076452593784c583c9f2c50" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 40047 } } peers { permanent_uuid: "6c61b31cfd744e88a5fedcdd8838a81d" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 35075 } } }
I20240501 14:08:50.010584 16669 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb4af6c1f076452593784c583c9f2c50 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:08:50.010640 16571 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb4af6c1f076452593784c583c9f2c50 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:08:50.013010 16439 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240501 14:08:50.013660 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:50.013772 16680 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eb4af6c1f076452593784c583c9f2c50: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 14:08:50.013826 16680 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240501 14:08:50.016100 16681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:50.016903 16682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:50.017751 16684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:50.018332 16439 server_base.cc:1034] running on GCE node
I20240501 14:08:50.018503 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:08:50.018544 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:08:50.018568 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572530018567 us; error 0 us; skew 500 ppm
I20240501 14:08:50.019579 16439 webserver.cc:460] Webserver started at http://127.16.13.193:38465/ using document root <none> and password file <none>
I20240501 14:08:50.019733 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:50.019783 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:50.019869 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:50.020288 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714572529860089-16439-0/minicluster-data/ts-0-root/instance:
uuid: "3c323b6364834038aca2b8d5e87071c4"
format_stamp: "Formatted at 2024-05-01 14:08:50 on dist-test-slave-mgxc"
I20240501 14:08:50.021669 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 14:08:50.022555 16689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:50.022841 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:50.022904 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714572529860089-16439-0/minicluster-data/ts-0-root
uuid: "3c323b6364834038aca2b8d5e87071c4"
format_stamp: "Formatted at 2024-05-01 14:08:50 on dist-test-slave-mgxc"
I20240501 14:08:50.022966 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714572529860089-16439-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714572529860089-16439-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714572529860089-16439-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:50.031739 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:50.032173 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:50.032581 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:50.033340 16439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:50.033391 16439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:50.033458 16439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:50.033490 16439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:50.039433 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.193:36085
I20240501 14:08:50.039477 16752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.193:36085 every 8 connection(s)
I20240501 14:08:50.040597 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:50.043998 16759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:50.044232 16762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:50.045105 16760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:50.045636 16754 heartbeater.cc:347] Connected to a master server at 127.16.13.252:35075
I20240501 14:08:50.045770 16754 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:50.046394 16754 heartbeater.cc:511] Master 127.16.13.252:35075 requested a full tablet report, sending...
I20240501 14:08:50.046501 16753 heartbeater.cc:347] Connected to a master server at 127.16.13.254:46433
I20240501 14:08:50.046592 16753 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:50.046803 16753 heartbeater.cc:511] Master 127.16.13.254:46433 requested a full tablet report, sending...
I20240501 14:08:50.045641 16439 server_base.cc:1034] running on GCE node
I20240501 14:08:50.047096 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:08:50.047127 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:08:50.047148 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572530047148 us; error 0 us; skew 500 ppm
I20240501 14:08:50.048301 16601 ts_manager.cc:194] Registered new tserver with Master: 3c323b6364834038aca2b8d5e87071c4 (127.16.13.193:36085)
I20240501 14:08:50.048591 16469 ts_manager.cc:194] Registered new tserver with Master: 3c323b6364834038aca2b8d5e87071c4 (127.16.13.193:36085)
I20240501 14:08:50.049228 16755 heartbeater.cc:347] Connected to a master server at 127.16.13.253:40047
I20240501 14:08:50.049368 16755 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:50.049371 16439 webserver.cc:460] Webserver started at http://127.16.13.194:43025/ using document root <none> and password file <none>
I20240501 14:08:50.049582 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:50.049628 16755 heartbeater.cc:511] Master 127.16.13.253:40047 requested a full tablet report, sending...
I20240501 14:08:50.049643 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:50.049813 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:50.050269 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714572529860089-16439-0/minicluster-data/ts-1-root/instance:
uuid: "51ba16800f1447fa81034ce650ba9d99"
format_stamp: "Formatted at 2024-05-01 14:08:50 on dist-test-slave-mgxc"
I20240501 14:08:50.050339 16526 ts_manager.cc:194] Registered new tserver with Master: 3c323b6364834038aca2b8d5e87071c4 (127.16.13.193:36085)
I20240501 14:08:50.051965 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 14:08:50.052979 16768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:50.053265 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:50.053324 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714572529860089-16439-0/minicluster-data/ts-1-root
uuid: "51ba16800f1447fa81034ce650ba9d99"
format_stamp: "Formatted at 2024-05-01 14:08:50 on dist-test-slave-mgxc"
I20240501 14:08:50.053386 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714572529860089-16439-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714572529860089-16439-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714572529860089-16439-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:50.076221 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:50.076633 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:50.077245 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:50.077751 16439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:50.077793 16439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:50.077826 16439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:50.077859 16439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:50.083009 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.194:41965
I20240501 14:08:50.083124 16831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.194:41965 every 8 connection(s)
I20240501 14:08:50.084057 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:50.086937 16838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:50.087826 16839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:50.089300 16841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:50.089973 16833 heartbeater.cc:347] Connected to a master server at 127.16.13.252:35075
I20240501 14:08:50.090072 16833 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:50.090092 16832 heartbeater.cc:347] Connected to a master server at 127.16.13.254:46433
I20240501 14:08:50.090183 16832 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:50.090286 16833 heartbeater.cc:511] Master 127.16.13.252:35075 requested a full tablet report, sending...
I20240501 14:08:50.090378 16832 heartbeater.cc:511] Master 127.16.13.254:46433 requested a full tablet report, sending...
I20240501 14:08:50.090924 16601 ts_manager.cc:194] Registered new tserver with Master: 51ba16800f1447fa81034ce650ba9d99 (127.16.13.194:41965)
I20240501 14:08:50.091001 16469 ts_manager.cc:194] Registered new tserver with Master: 51ba16800f1447fa81034ce650ba9d99 (127.16.13.194:41965)
I20240501 14:08:50.091513 16439 server_base.cc:1034] running on GCE node
I20240501 14:08:50.091702 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:08:50.091750 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:08:50.091771 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572530091771 us; error 0 us; skew 500 ppm
I20240501 14:08:50.092216 16834 heartbeater.cc:347] Connected to a master server at 127.16.13.253:40047
I20240501 14:08:50.092305 16834 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:50.092499 16834 heartbeater.cc:511] Master 127.16.13.253:40047 requested a full tablet report, sending...
I20240501 14:08:50.092891 16439 webserver.cc:460] Webserver started at http://127.16.13.195:40161/ using document root <none> and password file <none>
I20240501 14:08:50.093048 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:50.093091 16526 ts_manager.cc:194] Registered new tserver with Master: 51ba16800f1447fa81034ce650ba9d99 (127.16.13.194:41965)
I20240501 14:08:50.093101 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:50.093232 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:50.093677 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714572529860089-16439-0/minicluster-data/ts-2-root/instance:
uuid: "89b19d1b8a984a2380eae955e5acc28a"
format_stamp: "Formatted at 2024-05-01 14:08:50 on dist-test-slave-mgxc"
I20240501 14:08:50.095070 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 14:08:50.096006 16847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:50.096254 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:50.096305 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714572529860089-16439-0/minicluster-data/ts-2-root
uuid: "89b19d1b8a984a2380eae955e5acc28a"
format_stamp: "Formatted at 2024-05-01 14:08:50 on dist-test-slave-mgxc"
I20240501 14:08:50.096364 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714572529860089-16439-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714572529860089-16439-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714572529860089-16439-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:50.107620 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:50.107995 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:50.108287 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:50.108731 16439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:50.108769 16439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:50.108804 16439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:50.108836 16439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:50.114567 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.195:46799
I20240501 14:08:50.114615 16910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.195:46799 every 8 connection(s)
I20240501 14:08:50.120362 16912 heartbeater.cc:347] Connected to a master server at 127.16.13.252:35075
I20240501 14:08:50.120460 16912 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:50.120468 16911 heartbeater.cc:347] Connected to a master server at 127.16.13.254:46433
I20240501 14:08:50.120543 16911 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:50.120725 16911 heartbeater.cc:511] Master 127.16.13.254:46433 requested a full tablet report, sending...
I20240501 14:08:50.120725 16912 heartbeater.cc:511] Master 127.16.13.252:35075 requested a full tablet report, sending...
I20240501 14:08:50.121585 16601 ts_manager.cc:194] Registered new tserver with Master: 89b19d1b8a984a2380eae955e5acc28a (127.16.13.195:46799)
I20240501 14:08:50.121585 16469 ts_manager.cc:194] Registered new tserver with Master: 89b19d1b8a984a2380eae955e5acc28a (127.16.13.195:46799)
I20240501 14:08:50.121884 16913 heartbeater.cc:347] Connected to a master server at 127.16.13.253:40047
I20240501 14:08:50.121958 16913 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:50.122155 16913 heartbeater.cc:511] Master 127.16.13.253:40047 requested a full tablet report, sending...
I20240501 14:08:50.122749 16526 ts_manager.cc:194] Registered new tserver with Master: 89b19d1b8a984a2380eae955e5acc28a (127.16.13.195:46799)
I20240501 14:08:50.122865 16439 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007782272s
W20240501 14:08:50.159144 16458 tablet.cc:2369] T 00000000000000000000000000000000 P 0d83058ad11c489aba83f7bb5a48d430: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:08:50.220642 16642 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:50.220892 16642 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d83058ad11c489aba83f7bb5a48d430" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 46433 } } peers { permanent_uuid: "eb4af6c1f076452593784c583c9f2c50" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 40047 } } peers { permanent_uuid: "6c61b31cfd744e88a5fedcdd8838a81d" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 35075 } }
I20240501 14:08:50.221951 16642 leader_election.cc:288] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d83058ad11c489aba83f7bb5a48d430 (127.16.13.254:46433), eb4af6c1f076452593784c583c9f2c50 (127.16.13.253:40047)
I20240501 14:08:50.222482 16479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c61b31cfd744e88a5fedcdd8838a81d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d83058ad11c489aba83f7bb5a48d430" is_pre_election: true
I20240501 14:08:50.222486 16544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c61b31cfd744e88a5fedcdd8838a81d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb4af6c1f076452593784c583c9f2c50" is_pre_election: true
I20240501 14:08:50.222681 16544 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eb4af6c1f076452593784c583c9f2c50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c61b31cfd744e88a5fedcdd8838a81d in term 0.
I20240501 14:08:50.222679 16479 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0d83058ad11c489aba83f7bb5a48d430 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c61b31cfd744e88a5fedcdd8838a81d in term 0.
I20240501 14:08:50.223132 16587 leader_election.cc:302] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d83058ad11c489aba83f7bb5a48d430, 6c61b31cfd744e88a5fedcdd8838a81d; no voters:
I20240501 14:08:50.223383 16642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:50.223446 16642 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:50.223484 16642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:50.224692 16642 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d83058ad11c489aba83f7bb5a48d430" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 46433 } } peers { permanent_uuid: "eb4af6c1f076452593784c583c9f2c50" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 40047 } } peers { permanent_uuid: "6c61b31cfd744e88a5fedcdd8838a81d" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 35075 } }
I20240501 14:08:50.225165 16642 leader_election.cc:288] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d [CANDIDATE]: Term 1 election: Requested vote from peers 0d83058ad11c489aba83f7bb5a48d430 (127.16.13.254:46433), eb4af6c1f076452593784c583c9f2c50 (127.16.13.253:40047)
I20240501 14:08:50.225492 16479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c61b31cfd744e88a5fedcdd8838a81d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d83058ad11c489aba83f7bb5a48d430"
I20240501 14:08:50.225602 16544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c61b31cfd744e88a5fedcdd8838a81d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb4af6c1f076452593784c583c9f2c50"
I20240501 14:08:50.225612 16479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d83058ad11c489aba83f7bb5a48d430 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:50.225731 16544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb4af6c1f076452593784c583c9f2c50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:50.226408 16479 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0d83058ad11c489aba83f7bb5a48d430 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c61b31cfd744e88a5fedcdd8838a81d in term 1.
I20240501 14:08:50.226691 16587 leader_election.cc:302] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d83058ad11c489aba83f7bb5a48d430, 6c61b31cfd744e88a5fedcdd8838a81d; no voters:
I20240501 14:08:50.226783 16544 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eb4af6c1f076452593784c583c9f2c50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c61b31cfd744e88a5fedcdd8838a81d in term 1.
I20240501 14:08:50.226842 16642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:50.227034 16642 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d [term 1 LEADER]: Becoming Leader. State: Replica: 6c61b31cfd744e88a5fedcdd8838a81d, State: Running, Role: LEADER
I20240501 14:08:50.227329 16642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d83058ad11c489aba83f7bb5a48d430" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 46433 } } peers { permanent_uuid: "eb4af6c1f076452593784c583c9f2c50" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 40047 } } peers { permanent_uuid: "6c61b31cfd744e88a5fedcdd8838a81d" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 35075 } }
I20240501 14:08:50.228878 16919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c61b31cfd744e88a5fedcdd8838a81d. Latest consensus state: current_term: 1 leader_uuid: "6c61b31cfd744e88a5fedcdd8838a81d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d83058ad11c489aba83f7bb5a48d430" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 46433 } } peers { permanent_uuid: "eb4af6c1f076452593784c583c9f2c50" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 40047 } } peers { permanent_uuid: "6c61b31cfd744e88a5fedcdd8838a81d" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 35075 } } }
I20240501 14:08:50.228991 16919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d [sys.catalog]: This master's current role is: LEADER
I20240501 14:08:50.229342 16921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:08:50.230186 16921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:08:50.234293 16544 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eb4af6c1f076452593784c583c9f2c50 [term 1 FOLLOWER]: Refusing update from remote peer 6c61b31cfd744e88a5fedcdd8838a81d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:50.234293 16479 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d83058ad11c489aba83f7bb5a48d430 [term 1 FOLLOWER]: Refusing update from remote peer 6c61b31cfd744e88a5fedcdd8838a81d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:50.234808 16919 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d83058ad11c489aba83f7bb5a48d430" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 46433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:50.234952 16642 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb4af6c1f076452593784c583c9f2c50" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:50.236387 16669 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb4af6c1f076452593784c583c9f2c50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c61b31cfd744e88a5fedcdd8838a81d. Latest consensus state: current_term: 1 leader_uuid: "6c61b31cfd744e88a5fedcdd8838a81d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d83058ad11c489aba83f7bb5a48d430" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 46433 } } peers { permanent_uuid: "eb4af6c1f076452593784c583c9f2c50" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 40047 } } peers { permanent_uuid: "6c61b31cfd744e88a5fedcdd8838a81d" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 35075 } } }
I20240501 14:08:50.236393 16656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d83058ad11c489aba83f7bb5a48d430 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c61b31cfd744e88a5fedcdd8838a81d. Latest consensus state: current_term: 1 leader_uuid: "6c61b31cfd744e88a5fedcdd8838a81d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d83058ad11c489aba83f7bb5a48d430" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 46433 } } peers { permanent_uuid: "eb4af6c1f076452593784c583c9f2c50" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 40047 } } peers { permanent_uuid: "6c61b31cfd744e88a5fedcdd8838a81d" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 35075 } } }
I20240501 14:08:50.236565 16669 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb4af6c1f076452593784c583c9f2c50 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:08:50.236575 16656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d83058ad11c489aba83f7bb5a48d430 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:08:50.238065 16927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c61b31cfd744e88a5fedcdd8838a81d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d83058ad11c489aba83f7bb5a48d430" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 46433 } } peers { permanent_uuid: "eb4af6c1f076452593784c583c9f2c50" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 40047 } } peers { permanent_uuid: "6c61b31cfd744e88a5fedcdd8838a81d" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 35075 } } }
I20240501 14:08:50.238116 16919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c61b31cfd744e88a5fedcdd8838a81d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d83058ad11c489aba83f7bb5a48d430" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 46433 } } peers { permanent_uuid: "eb4af6c1f076452593784c583c9f2c50" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 40047 } } peers { permanent_uuid: "6c61b31cfd744e88a5fedcdd8838a81d" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 35075 } } }
I20240501 14:08:50.238174 16927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d [sys.catalog]: This master's current role is: LEADER
I20240501 14:08:50.238224 16919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d [sys.catalog]: This master's current role is: LEADER
I20240501 14:08:50.238178 16656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d83058ad11c489aba83f7bb5a48d430 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c61b31cfd744e88a5fedcdd8838a81d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d83058ad11c489aba83f7bb5a48d430" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 46433 } } peers { permanent_uuid: "eb4af6c1f076452593784c583c9f2c50" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 40047 } } peers { permanent_uuid: "6c61b31cfd744e88a5fedcdd8838a81d" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 35075 } } }
I20240501 14:08:50.238408 16656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d83058ad11c489aba83f7bb5a48d430 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:08:50.238497 16669 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb4af6c1f076452593784c583c9f2c50 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c61b31cfd744e88a5fedcdd8838a81d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d83058ad11c489aba83f7bb5a48d430" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 46433 } } peers { permanent_uuid: "eb4af6c1f076452593784c583c9f2c50" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 40047 } } peers { permanent_uuid: "6c61b31cfd744e88a5fedcdd8838a81d" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 35075 } } }
I20240501 14:08:50.238600 16669 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb4af6c1f076452593784c583c9f2c50 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:08:50.238693 16924 mvcc.cc:205] Tried to move back new op lower bound from 7022889083834281984 to 7022889083813371904. Current Snapshot: MvccSnapshot[applied={T|T < 7022889083834281984}]
I20240501 14:08:50.240183 16921 catalog_manager.cc:1347] Generated new cluster ID: 24d37a51840340c7825da1b31e226737
I20240501 14:08:50.240238 16921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:08:50.257968 16921 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:08:50.258972 16921 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:08:50.268106 16921 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d: Generated new TSK 0
I20240501 14:08:50.268900 16921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:08:51.001569 16667 catalog_manager.cc:1259] Loaded cluster ID: 24d37a51840340c7825da1b31e226737
I20240501 14:08:51.001672 16667 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0d83058ad11c489aba83f7bb5a48d430: loading cluster ID for follower catalog manager: success
I20240501 14:08:51.003444 16667 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0d83058ad11c489aba83f7bb5a48d430: acquiring CA information for follower catalog manager: success
I20240501 14:08:51.004241 16667 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0d83058ad11c489aba83f7bb5a48d430: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 14:08:51.014536 16680 catalog_manager.cc:1259] Loaded cluster ID: 24d37a51840340c7825da1b31e226737
I20240501 14:08:51.014608 16680 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eb4af6c1f076452593784c583c9f2c50: loading cluster ID for follower catalog manager: success
I20240501 14:08:51.015707 16680 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eb4af6c1f076452593784c583c9f2c50: acquiring CA information for follower catalog manager: success
I20240501 14:08:51.016285 16680 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eb4af6c1f076452593784c583c9f2c50: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 14:08:51.051160 16601 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43906
I20240501 14:08:51.092816 16601 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43916
I20240501 14:08:51.123905 16601 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43920
I20240501 14:08:51.989377 16652 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:08:52.052371 16754 heartbeater.cc:503] Master 127.16.13.252:35075 was elected leader, sending a full tablet report...
I20240501 14:08:52.093704 16833 heartbeater.cc:503] Master 127.16.13.252:35075 was elected leader, sending a full tablet report...
I20240501 14:08:52.124794 16912 heartbeater.cc:503] Master 127.16.13.252:35075 was elected leader, sending a full tablet report...
I20240501 14:08:52.389335 16439 test_util.cc:268] Using random seed: 112989650
I20240501 14:08:52.404964 16601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40376:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:08:52.422798 16716 tablet_service.cc:1467] Processing CreateTablet for tablet 5e565edac5c74c22b04674e4b686559b (DEFAULT_TABLE table=test-workload [id=94467bebefda4ecd9f740e5a557cda60]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240501 14:08:52.422816 16717 tablet_service.cc:1467] Processing CreateTablet for tablet de8863481c6d488a93dd41446e120393 (DEFAULT_TABLE table=test-workload [id=94467bebefda4ecd9f740e5a557cda60]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240501 14:08:52.423306 16716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e565edac5c74c22b04674e4b686559b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:52.423353 16796 tablet_service.cc:1467] Processing CreateTablet for tablet 03e6e9b65e9941eb8eee6ae31330e2a6 (DEFAULT_TABLE table=test-workload [id=94467bebefda4ecd9f740e5a557cda60]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240501 14:08:52.423485 16717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de8863481c6d488a93dd41446e120393. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:52.422798 16875 tablet_service.cc:1467] Processing CreateTablet for tablet 490f812b37904285934bd24aecd6db12 (DEFAULT_TABLE table=test-workload [id=94467bebefda4ecd9f740e5a557cda60]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240501 14:08:52.423828 16875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 490f812b37904285934bd24aecd6db12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:52.425627 16958 tablet_bootstrap.cc:492] T de8863481c6d488a93dd41446e120393 P 3c323b6364834038aca2b8d5e87071c4: Bootstrap starting.
I20240501 14:08:52.425758 16959 tablet_bootstrap.cc:492] T 490f812b37904285934bd24aecd6db12 P 89b19d1b8a984a2380eae955e5acc28a: Bootstrap starting.
I20240501 14:08:52.423596 16796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e6e9b65e9941eb8eee6ae31330e2a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:52.426635 16959 tablet_bootstrap.cc:654] T 490f812b37904285934bd24aecd6db12 P 89b19d1b8a984a2380eae955e5acc28a: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:52.426672 16958 tablet_bootstrap.cc:654] T de8863481c6d488a93dd41446e120393 P 3c323b6364834038aca2b8d5e87071c4: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:52.427525 16959 tablet_bootstrap.cc:492] T 490f812b37904285934bd24aecd6db12 P 89b19d1b8a984a2380eae955e5acc28a: No bootstrap required, opened a new log
I20240501 14:08:52.427592 16959 ts_tablet_manager.cc:1367] T 490f812b37904285934bd24aecd6db12 P 89b19d1b8a984a2380eae955e5acc28a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:52.427731 16958 tablet_bootstrap.cc:492] T de8863481c6d488a93dd41446e120393 P 3c323b6364834038aca2b8d5e87071c4: No bootstrap required, opened a new log
I20240501 14:08:52.427804 16958 ts_tablet_manager.cc:1367] T de8863481c6d488a93dd41446e120393 P 3c323b6364834038aca2b8d5e87071c4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:52.427909 16959 raft_consensus.cc:348] T 490f812b37904285934bd24aecd6db12 P 89b19d1b8a984a2380eae955e5acc28a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b19d1b8a984a2380eae955e5acc28a" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 46799 } }
I20240501 14:08:52.428026 16959 raft_consensus.cc:374] T 490f812b37904285934bd24aecd6db12 P 89b19d1b8a984a2380eae955e5acc28a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:52.428020 16962 tablet_bootstrap.cc:492] T 03e6e9b65e9941eb8eee6ae31330e2a6 P 51ba16800f1447fa81034ce650ba9d99: Bootstrap starting.
I20240501 14:08:52.428108 16959 raft_consensus.cc:729] T 490f812b37904285934bd24aecd6db12 P 89b19d1b8a984a2380eae955e5acc28a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89b19d1b8a984a2380eae955e5acc28a, State: Initialized, Role: FOLLOWER
I20240501 14:08:52.428231 16959 consensus_queue.cc:260] T 490f812b37904285934bd24aecd6db12 P 89b19d1b8a984a2380eae955e5acc28a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b19d1b8a984a2380eae955e5acc28a" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 46799 } }
I20240501 14:08:52.428254 16958 raft_consensus.cc:348] T de8863481c6d488a93dd41446e120393 P 3c323b6364834038aca2b8d5e87071c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c323b6364834038aca2b8d5e87071c4" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 36085 } }
I20240501 14:08:52.428352 16959 raft_consensus.cc:388] T 490f812b37904285934bd24aecd6db12 P 89b19d1b8a984a2380eae955e5acc28a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:08:52.428393 16958 raft_consensus.cc:374] T de8863481c6d488a93dd41446e120393 P 3c323b6364834038aca2b8d5e87071c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:52.428431 16959 raft_consensus.cc:482] T 490f812b37904285934bd24aecd6db12 P 89b19d1b8a984a2380eae955e5acc28a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:08:52.428452 16958 raft_consensus.cc:729] T de8863481c6d488a93dd41446e120393 P 3c323b6364834038aca2b8d5e87071c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c323b6364834038aca2b8d5e87071c4, State: Initialized, Role: FOLLOWER
I20240501 14:08:52.428498 16959 raft_consensus.cc:3037] T 490f812b37904285934bd24aecd6db12 P 89b19d1b8a984a2380eae955e5acc28a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:52.428612 16958 consensus_queue.cc:260] T de8863481c6d488a93dd41446e120393 P 3c323b6364834038aca2b8d5e87071c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c323b6364834038aca2b8d5e87071c4" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 36085 } }
I20240501 14:08:52.428740 16958 raft_consensus.cc:388] T de8863481c6d488a93dd41446e120393 P 3c323b6364834038aca2b8d5e87071c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:08:52.428790 16958 raft_consensus.cc:482] T de8863481c6d488a93dd41446e120393 P 3c323b6364834038aca2b8d5e87071c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:08:52.428844 16958 raft_consensus.cc:3037] T de8863481c6d488a93dd41446e120393 P 3c323b6364834038aca2b8d5e87071c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:52.428879 16962 tablet_bootstrap.cc:654] T 03e6e9b65e9941eb8eee6ae31330e2a6 P 51ba16800f1447fa81034ce650ba9d99: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:52.429400 16959 raft_consensus.cc:504] T 490f812b37904285934bd24aecd6db12 P 89b19d1b8a984a2380eae955e5acc28a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b19d1b8a984a2380eae955e5acc28a" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 46799 } }
I20240501 14:08:52.429533 16959 leader_election.cc:302] T 490f812b37904285934bd24aecd6db12 P 89b19d1b8a984a2380eae955e5acc28a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89b19d1b8a984a2380eae955e5acc28a; no voters:
I20240501 14:08:52.429633 16958 raft_consensus.cc:504] T de8863481c6d488a93dd41446e120393 P 3c323b6364834038aca2b8d5e87071c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c323b6364834038aca2b8d5e87071c4" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 36085 } }
I20240501 14:08:52.429703 16959 leader_election.cc:288] T 490f812b37904285934bd24aecd6db12 P 89b19d1b8a984a2380eae955e5acc28a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:08:52.429776 16958 leader_election.cc:302] T de8863481c6d488a93dd41446e120393 P 3c323b6364834038aca2b8d5e87071c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c323b6364834038aca2b8d5e87071c4; no voters:
I20240501 14:08:52.429787 16964 raft_consensus.cc:2781] T 490f812b37904285934bd24aecd6db12 P 89b19d1b8a984a2380eae955e5acc28a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:52.429989 16958 leader_election.cc:288] T de8863481c6d488a93dd41446e120393 P 3c323b6364834038aca2b8d5e87071c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:08:52.430013 16964 raft_consensus.cc:686] T 490f812b37904285934bd24aecd6db12 P 89b19d1b8a984a2380eae955e5acc28a [term 1 LEADER]: Becoming Leader. State: Replica: 89b19d1b8a984a2380eae955e5acc28a, State: Running, Role: LEADER
I20240501 14:08:52.430099 16959 ts_tablet_manager.cc:1398] T 490f812b37904285934bd24aecd6db12 P 89b19d1b8a984a2380eae955e5acc28a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:08:52.430191 16964 consensus_queue.cc:237] T 490f812b37904285934bd24aecd6db12 P 89b19d1b8a984a2380eae955e5acc28a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b19d1b8a984a2380eae955e5acc28a" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 46799 } }
I20240501 14:08:52.430294 16958 ts_tablet_manager.cc:1398] T de8863481c6d488a93dd41446e120393 P 3c323b6364834038aca2b8d5e87071c4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:08:52.432420 16962 tablet_bootstrap.cc:492] T 03e6e9b65e9941eb8eee6ae31330e2a6 P 51ba16800f1447fa81034ce650ba9d99: No bootstrap required, opened a new log
I20240501 14:08:52.432503 16962 ts_tablet_manager.cc:1367] T 03e6e9b65e9941eb8eee6ae31330e2a6 P 51ba16800f1447fa81034ce650ba9d99: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:08:52.432858 16962 raft_consensus.cc:348] T 03e6e9b65e9941eb8eee6ae31330e2a6 P 51ba16800f1447fa81034ce650ba9d99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ba16800f1447fa81034ce650ba9d99" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 41965 } }
I20240501 14:08:52.432941 16962 raft_consensus.cc:374] T 03e6e9b65e9941eb8eee6ae31330e2a6 P 51ba16800f1447fa81034ce650ba9d99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:52.432972 16962 raft_consensus.cc:729] T 03e6e9b65e9941eb8eee6ae31330e2a6 P 51ba16800f1447fa81034ce650ba9d99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51ba16800f1447fa81034ce650ba9d99, State: Initialized, Role: FOLLOWER
I20240501 14:08:52.433068 16962 consensus_queue.cc:260] T 03e6e9b65e9941eb8eee6ae31330e2a6 P 51ba16800f1447fa81034ce650ba9d99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ba16800f1447fa81034ce650ba9d99" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 41965 } }
I20240501 14:08:52.433135 16962 raft_consensus.cc:388] T 03e6e9b65e9941eb8eee6ae31330e2a6 P 51ba16800f1447fa81034ce650ba9d99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:08:52.433166 16962 raft_consensus.cc:482] T 03e6e9b65e9941eb8eee6ae31330e2a6 P 51ba16800f1447fa81034ce650ba9d99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:08:52.433203 16962 raft_consensus.cc:3037] T 03e6e9b65e9941eb8eee6ae31330e2a6 P 51ba16800f1447fa81034ce650ba9d99 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:52.434012 16962 raft_consensus.cc:504] T 03e6e9b65e9941eb8eee6ae31330e2a6 P 51ba16800f1447fa81034ce650ba9d99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ba16800f1447fa81034ce650ba9d99" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 41965 } }
I20240501 14:08:52.434121 16962 leader_election.cc:302] T 03e6e9b65e9941eb8eee6ae31330e2a6 P 51ba16800f1447fa81034ce650ba9d99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51ba16800f1447fa81034ce650ba9d99; no voters:
I20240501 14:08:52.434233 16958 tablet_bootstrap.cc:492] T 5e565edac5c74c22b04674e4b686559b P 3c323b6364834038aca2b8d5e87071c4: Bootstrap starting.
I20240501 14:08:52.434304 16962 leader_election.cc:288] T 03e6e9b65e9941eb8eee6ae31330e2a6 P 51ba16800f1447fa81034ce650ba9d99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:08:52.434473 16971 raft_consensus.cc:2781] T 03e6e9b65e9941eb8eee6ae31330e2a6 P 51ba16800f1447fa81034ce650ba9d99 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:52.434584 16962 ts_tablet_manager.cc:1398] T 03e6e9b65e9941eb8eee6ae31330e2a6 P 51ba16800f1447fa81034ce650ba9d99: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:52.434662 16971 raft_consensus.cc:686] T 03e6e9b65e9941eb8eee6ae31330e2a6 P 51ba16800f1447fa81034ce650ba9d99 [term 1 LEADER]: Becoming Leader. State: Replica: 51ba16800f1447fa81034ce650ba9d99, State: Running, Role: LEADER
I20240501 14:08:52.434774 16971 consensus_queue.cc:237] T 03e6e9b65e9941eb8eee6ae31330e2a6 P 51ba16800f1447fa81034ce650ba9d99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ba16800f1447fa81034ce650ba9d99" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 41965 } }
I20240501 14:08:52.435241 16958 tablet_bootstrap.cc:654] T 5e565edac5c74c22b04674e4b686559b P 3c323b6364834038aca2b8d5e87071c4: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:52.435660 16601 catalog_manager.cc:5484] T 490f812b37904285934bd24aecd6db12 P 89b19d1b8a984a2380eae955e5acc28a reported cstate change: term changed from 0 to 1, leader changed from <none> to 89b19d1b8a984a2380eae955e5acc28a (127.16.13.195). New cstate: current_term: 1 leader_uuid: "89b19d1b8a984a2380eae955e5acc28a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b19d1b8a984a2380eae955e5acc28a" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 46799 } health_report { overall_health: HEALTHY } } }
I20240501 14:08:52.435864 16968 raft_consensus.cc:2781] T de8863481c6d488a93dd41446e120393 P 3c323b6364834038aca2b8d5e87071c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:52.436053 16968 raft_consensus.cc:686] T de8863481c6d488a93dd41446e120393 P 3c323b6364834038aca2b8d5e87071c4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c323b6364834038aca2b8d5e87071c4, State: Running, Role: LEADER
I20240501 14:08:52.436154 16968 consensus_queue.cc:237] T de8863481c6d488a93dd41446e120393 P 3c323b6364834038aca2b8d5e87071c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c323b6364834038aca2b8d5e87071c4" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 36085 } }
I20240501 14:08:52.438184 16599 catalog_manager.cc:5484] T de8863481c6d488a93dd41446e120393 P 3c323b6364834038aca2b8d5e87071c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c323b6364834038aca2b8d5e87071c4 (127.16.13.193). New cstate: current_term: 1 leader_uuid: "3c323b6364834038aca2b8d5e87071c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c323b6364834038aca2b8d5e87071c4" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 36085 } health_report { overall_health: HEALTHY } } }
I20240501 14:08:52.440742 16958 tablet_bootstrap.cc:492] T 5e565edac5c74c22b04674e4b686559b P 3c323b6364834038aca2b8d5e87071c4: No bootstrap required, opened a new log
I20240501 14:08:52.440896 16958 ts_tablet_manager.cc:1367] T 5e565edac5c74c22b04674e4b686559b P 3c323b6364834038aca2b8d5e87071c4: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:08:52.435777 16600 catalog_manager.cc:5484] T 03e6e9b65e9941eb8eee6ae31330e2a6 P 51ba16800f1447fa81034ce650ba9d99 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51ba16800f1447fa81034ce650ba9d99 (127.16.13.194). New cstate: current_term: 1 leader_uuid: "51ba16800f1447fa81034ce650ba9d99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ba16800f1447fa81034ce650ba9d99" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 41965 } health_report { overall_health: HEALTHY } } }
I20240501 14:08:52.441315 16958 raft_consensus.cc:348] T 5e565edac5c74c22b04674e4b686559b P 3c323b6364834038aca2b8d5e87071c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c323b6364834038aca2b8d5e87071c4" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 36085 } }
I20240501 14:08:52.441573 16958 raft_consensus.cc:374] T 5e565edac5c74c22b04674e4b686559b P 3c323b6364834038aca2b8d5e87071c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:52.441682 16958 raft_consensus.cc:729] T 5e565edac5c74c22b04674e4b686559b P 3c323b6364834038aca2b8d5e87071c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c323b6364834038aca2b8d5e87071c4, State: Initialized, Role: FOLLOWER
I20240501 14:08:52.441886 16958 consensus_queue.cc:260] T 5e565edac5c74c22b04674e4b686559b P 3c323b6364834038aca2b8d5e87071c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c323b6364834038aca2b8d5e87071c4" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 36085 } }
I20240501 14:08:52.442039 16958 raft_consensus.cc:388] T 5e565edac5c74c22b04674e4b686559b P 3c323b6364834038aca2b8d5e87071c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:08:52.442118 16958 raft_consensus.cc:482] T 5e565edac5c74c22b04674e4b686559b P 3c323b6364834038aca2b8d5e87071c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:08:52.442211 16958 raft_consensus.cc:3037] T 5e565edac5c74c22b04674e4b686559b P 3c323b6364834038aca2b8d5e87071c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:52.444873 16958 raft_consensus.cc:504] T 5e565edac5c74c22b04674e4b686559b P 3c323b6364834038aca2b8d5e87071c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c323b6364834038aca2b8d5e87071c4" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 36085 } }
I20240501 14:08:52.445150 16958 leader_election.cc:302] T 5e565edac5c74c22b04674e4b686559b P 3c323b6364834038aca2b8d5e87071c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c323b6364834038aca2b8d5e87071c4; no voters:
I20240501 14:08:52.445259 16958 leader_election.cc:288] T 5e565edac5c74c22b04674e4b686559b P 3c323b6364834038aca2b8d5e87071c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:08:52.445452 16958 ts_tablet_manager.cc:1398] T 5e565edac5c74c22b04674e4b686559b P 3c323b6364834038aca2b8d5e87071c4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:08:52.447718 16968 raft_consensus.cc:2781] T 5e565edac5c74c22b04674e4b686559b P 3c323b6364834038aca2b8d5e87071c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:52.447810 16968 raft_consensus.cc:686] T 5e565edac5c74c22b04674e4b686559b P 3c323b6364834038aca2b8d5e87071c4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c323b6364834038aca2b8d5e87071c4, State: Running, Role: LEADER
I20240501 14:08:52.447947 16968 consensus_queue.cc:237] T 5e565edac5c74c22b04674e4b686559b P 3c323b6364834038aca2b8d5e87071c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c323b6364834038aca2b8d5e87071c4" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 36085 } }
I20240501 14:08:52.449620 16599 catalog_manager.cc:5484] T 5e565edac5c74c22b04674e4b686559b P 3c323b6364834038aca2b8d5e87071c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c323b6364834038aca2b8d5e87071c4 (127.16.13.193). New cstate: current_term: 1 leader_uuid: "3c323b6364834038aca2b8d5e87071c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c323b6364834038aca2b8d5e87071c4" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 36085 } health_report { overall_health: HEALTHY } } }
I20240501 14:08:52.460474 16439 tablet_server.cc:178] TabletServer@127.16.13.193:0 shutting down...
I20240501 14:08:52.464797 16439 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:08:52.465040 16439 tablet_replica.cc:330] T 5e565edac5c74c22b04674e4b686559b P 3c323b6364834038aca2b8d5e87071c4: stopping tablet replica
I20240501 14:08:52.465267 16439 raft_consensus.cc:2229] T 5e565edac5c74c22b04674e4b686559b P 3c323b6364834038aca2b8d5e87071c4 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:08:52.465381 16439 raft_consensus.cc:2258] T 5e565edac5c74c22b04674e4b686559b P 3c323b6364834038aca2b8d5e87071c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:52.466329 16439 tablet_replica.cc:330] T de8863481c6d488a93dd41446e120393 P 3c323b6364834038aca2b8d5e87071c4: stopping tablet replica
I20240501 14:08:52.466405 16439 raft_consensus.cc:2229] T de8863481c6d488a93dd41446e120393 P 3c323b6364834038aca2b8d5e87071c4 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:08:52.466478 16439 raft_consensus.cc:2258] T de8863481c6d488a93dd41446e120393 P 3c323b6364834038aca2b8d5e87071c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:52.480170 16439 tablet_server.cc:194] TabletServer@127.16.13.193:0 shutdown complete.
I20240501 14:08:52.483712 16439 tablet_server.cc:178] TabletServer@127.16.13.194:0 shutting down...
I20240501 14:08:52.487699 16439 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:08:52.488126 16439 tablet_replica.cc:330] T 03e6e9b65e9941eb8eee6ae31330e2a6 P 51ba16800f1447fa81034ce650ba9d99: stopping tablet replica
I20240501 14:08:52.488232 16439 raft_consensus.cc:2229] T 03e6e9b65e9941eb8eee6ae31330e2a6 P 51ba16800f1447fa81034ce650ba9d99 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:08:52.488328 16439 raft_consensus.cc:2258] T 03e6e9b65e9941eb8eee6ae31330e2a6 P 51ba16800f1447fa81034ce650ba9d99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:52.492033 16439 tablet_server.cc:194] TabletServer@127.16.13.194:0 shutdown complete.
I20240501 14:08:52.493649 16439 tablet_server.cc:178] TabletServer@127.16.13.195:0 shutting down...
I20240501 14:08:52.497799 16439 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:08:52.498019 16439 tablet_replica.cc:330] T 490f812b37904285934bd24aecd6db12 P 89b19d1b8a984a2380eae955e5acc28a: stopping tablet replica
I20240501 14:08:52.498131 16439 raft_consensus.cc:2229] T 490f812b37904285934bd24aecd6db12 P 89b19d1b8a984a2380eae955e5acc28a [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:08:52.498239 16439 raft_consensus.cc:2258] T 490f812b37904285934bd24aecd6db12 P 89b19d1b8a984a2380eae955e5acc28a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:52.511096 16439 tablet_server.cc:194] TabletServer@127.16.13.195:0 shutdown complete.
I20240501 14:08:52.512974 16439 master.cc:537] Master@127.16.13.254:46433 shutting down...
I20240501 14:08:52.516429 16439 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0d83058ad11c489aba83f7bb5a48d430 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:08:52.516551 16439 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0d83058ad11c489aba83f7bb5a48d430 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:52.516603 16439 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0d83058ad11c489aba83f7bb5a48d430: stopping tablet replica
I20240501 14:08:52.529335 16439 master.cc:552] Master@127.16.13.254:46433 shutdown complete.
I20240501 14:08:52.532136 16439 master.cc:537] Master@127.16.13.253:40047 shutting down...
I20240501 14:08:52.535339 16439 raft_consensus.cc:2229] T 00000000000000000000000000000000 P eb4af6c1f076452593784c583c9f2c50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:08:52.535470 16439 raft_consensus.cc:2258] T 00000000000000000000000000000000 P eb4af6c1f076452593784c583c9f2c50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:52.535519 16439 tablet_replica.cc:330] T 00000000000000000000000000000000 P eb4af6c1f076452593784c583c9f2c50: stopping tablet replica
I20240501 14:08:52.548110 16439 master.cc:552] Master@127.16.13.253:40047 shutdown complete.
I20240501 14:08:52.550515 16439 master.cc:537] Master@127.16.13.252:35075 shutting down...
I20240501 14:08:52.554426 16439 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:08:52.554675 16439 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:52.554729 16439 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6c61b31cfd744e88a5fedcdd8838a81d: stopping tablet replica
I20240501 14:08:52.567808 16439 master.cc:552] Master@127.16.13.252:35075 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (2706 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20240501 14:08:52.575793 16439 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.13.254:42089
I20240501 14:08:52.576172 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:52.577950 16982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:52.578780 16983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:52.579589 16985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:52.580173 16439 server_base.cc:1034] running on GCE node
I20240501 14:08:52.580314 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:08:52.580354 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:08:52.580369 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572532580369 us; error 0 us; skew 500 ppm
I20240501 14:08:52.581347 16439 webserver.cc:460] Webserver started at http://127.16.13.254:41867/ using document root <none> and password file <none>
I20240501 14:08:52.581519 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:52.581583 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:52.581668 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:52.582152 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714572529860089-16439-0/minicluster-data/master-0-root/instance:
uuid: "1aa9529348184ff1b8e0942957d9d84c"
format_stamp: "Formatted at 2024-05-01 14:08:52 on dist-test-slave-mgxc"
I20240501 14:08:52.583518 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240501 14:08:52.584519 16990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:52.584807 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240501 14:08:52.584865 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714572529860089-16439-0/minicluster-data/master-0-root
uuid: "1aa9529348184ff1b8e0942957d9d84c"
format_stamp: "Formatted at 2024-05-01 14:08:52 on dist-test-slave-mgxc"
I20240501 14:08:52.584914 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714572529860089-16439-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714572529860089-16439-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714572529860089-16439-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:52.607492 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:52.607826 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:52.611466 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.254:42089
I20240501 14:08:52.614168 17042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.254:42089 every 8 connection(s)
I20240501 14:08:52.614579 17043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:52.616329 17043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1aa9529348184ff1b8e0942957d9d84c: Bootstrap starting.
I20240501 14:08:52.617033 17043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1aa9529348184ff1b8e0942957d9d84c: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:52.617964 17043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1aa9529348184ff1b8e0942957d9d84c: No bootstrap required, opened a new log
I20240501 14:08:52.618286 17043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1aa9529348184ff1b8e0942957d9d84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa9529348184ff1b8e0942957d9d84c" member_type: VOTER }
I20240501 14:08:52.618368 17043 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1aa9529348184ff1b8e0942957d9d84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:52.618388 17043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1aa9529348184ff1b8e0942957d9d84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aa9529348184ff1b8e0942957d9d84c, State: Initialized, Role: FOLLOWER
I20240501 14:08:52.618511 17043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1aa9529348184ff1b8e0942957d9d84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa9529348184ff1b8e0942957d9d84c" member_type: VOTER }
I20240501 14:08:52.618587 17043 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1aa9529348184ff1b8e0942957d9d84c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:08:52.618611 17043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1aa9529348184ff1b8e0942957d9d84c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:08:52.618639 17043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1aa9529348184ff1b8e0942957d9d84c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:52.619210 17043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1aa9529348184ff1b8e0942957d9d84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa9529348184ff1b8e0942957d9d84c" member_type: VOTER }
I20240501 14:08:52.619319 17043 leader_election.cc:302] T 00000000000000000000000000000000 P 1aa9529348184ff1b8e0942957d9d84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1aa9529348184ff1b8e0942957d9d84c; no voters:
I20240501 14:08:52.619452 17043 leader_election.cc:288] T 00000000000000000000000000000000 P 1aa9529348184ff1b8e0942957d9d84c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:08:52.619575 17046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1aa9529348184ff1b8e0942957d9d84c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:52.619763 17046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1aa9529348184ff1b8e0942957d9d84c [term 1 LEADER]: Becoming Leader. State: Replica: 1aa9529348184ff1b8e0942957d9d84c, State: Running, Role: LEADER
I20240501 14:08:52.619827 17043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1aa9529348184ff1b8e0942957d9d84c [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:08:52.619882 17046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1aa9529348184ff1b8e0942957d9d84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa9529348184ff1b8e0942957d9d84c" member_type: VOTER }
I20240501 14:08:52.620384 17047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1aa9529348184ff1b8e0942957d9d84c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1aa9529348184ff1b8e0942957d9d84c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa9529348184ff1b8e0942957d9d84c" member_type: VOTER } }
I20240501 14:08:52.620513 17047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1aa9529348184ff1b8e0942957d9d84c [sys.catalog]: This master's current role is: LEADER
I20240501 14:08:52.620411 17048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1aa9529348184ff1b8e0942957d9d84c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1aa9529348184ff1b8e0942957d9d84c. Latest consensus state: current_term: 1 leader_uuid: "1aa9529348184ff1b8e0942957d9d84c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa9529348184ff1b8e0942957d9d84c" member_type: VOTER } }
I20240501 14:08:52.620718 17048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1aa9529348184ff1b8e0942957d9d84c [sys.catalog]: This master's current role is: LEADER
I20240501 14:08:52.620754 17053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:08:52.621563 17053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:08:52.621768 16439 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 14:08:52.623229 17053 catalog_manager.cc:1347] Generated new cluster ID: 45a7af9b3dc6458a988e11feb9cdbc9f
I20240501 14:08:52.623288 17053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:08:52.635058 17053 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:08:52.635650 17053 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:08:52.639853 17053 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1aa9529348184ff1b8e0942957d9d84c: Generated new TSK 0
I20240501 14:08:52.640043 17053 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:08:52.654003 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:52.655964 17064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:52.656656 17065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:52.657373 17067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:52.658027 16439 server_base.cc:1034] running on GCE node
I20240501 14:08:52.658213 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:08:52.658257 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:08:52.658278 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572532658278 us; error 0 us; skew 500 ppm
I20240501 14:08:52.659132 16439 webserver.cc:460] Webserver started at http://127.16.13.193:35201/ using document root <none> and password file <none>
I20240501 14:08:52.659296 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:52.659349 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:52.659430 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:52.659833 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714572529860089-16439-0/minicluster-data/ts-0-root/instance:
uuid: "694d2b1e07b84029a6232d1bbfba3be6"
format_stamp: "Formatted at 2024-05-01 14:08:52 on dist-test-slave-mgxc"
I20240501 14:08:52.661286 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:52.662122 17072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:52.662396 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:52.662472 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714572529860089-16439-0/minicluster-data/ts-0-root
uuid: "694d2b1e07b84029a6232d1bbfba3be6"
format_stamp: "Formatted at 2024-05-01 14:08:52 on dist-test-slave-mgxc"
I20240501 14:08:52.662540 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714572529860089-16439-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714572529860089-16439-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714572529860089-16439-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:52.682644 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:52.682998 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:52.683298 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:52.683730 16439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:52.683768 16439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:52.683816 16439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:52.683848 16439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:52.687932 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.193:33287
I20240501 14:08:52.687958 17135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.193:33287 every 8 connection(s)
I20240501 14:08:52.688652 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:52.690721 17140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:52.691550 17141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:52.692103 17143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:52.692845 16439 server_base.cc:1034] running on GCE node
I20240501 14:08:52.693015 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:08:52.693058 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:08:52.693100 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572532693099 us; error 0 us; skew 500 ppm
I20240501 14:08:52.693257 17136 heartbeater.cc:347] Connected to a master server at 127.16.13.254:42089
I20240501 14:08:52.693333 17136 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:52.693513 17136 heartbeater.cc:511] Master 127.16.13.254:42089 requested a full tablet report, sending...
I20240501 14:08:52.694101 17007 ts_manager.cc:194] Registered new tserver with Master: 694d2b1e07b84029a6232d1bbfba3be6 (127.16.13.193:33287)
I20240501 14:08:52.694176 16439 webserver.cc:460] Webserver started at http://127.16.13.194:34987/ using document root <none> and password file <none>
I20240501 14:08:52.694353 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:52.694407 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:52.694485 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:52.694873 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714572529860089-16439-0/minicluster-data/ts-1-root/instance:
uuid: "a4a6589543cb487b81dd6f0ec0a85fcb"
format_stamp: "Formatted at 2024-05-01 14:08:52 on dist-test-slave-mgxc"
I20240501 14:08:52.694880 17007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60008
I20240501 14:08:52.696521 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 14:08:52.697376 17148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:52.697619 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:52.697685 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714572529860089-16439-0/minicluster-data/ts-1-root
uuid: "a4a6589543cb487b81dd6f0ec0a85fcb"
format_stamp: "Formatted at 2024-05-01 14:08:52 on dist-test-slave-mgxc"
I20240501 14:08:52.697751 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714572529860089-16439-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714572529860089-16439-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714572529860089-16439-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:52.704087 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:52.704416 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:52.704690 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:52.705111 16439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:52.705155 16439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:52.705204 16439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:52.705237 16439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:52.711513 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.194:32803
I20240501 14:08:52.711539 17211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.194:32803 every 8 connection(s)
I20240501 14:08:52.712517 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:52.714555 17215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:52.715319 17216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:52.716114 17218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:52.716651 16439 server_base.cc:1034] running on GCE node
I20240501 14:08:52.716845 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:08:52.716885 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:08:52.716908 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572532716907 us; error 0 us; skew 500 ppm
I20240501 14:08:52.717001 17212 heartbeater.cc:347] Connected to a master server at 127.16.13.254:42089
I20240501 14:08:52.717078 17212 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:52.717280 17212 heartbeater.cc:511] Master 127.16.13.254:42089 requested a full tablet report, sending...
I20240501 14:08:52.717877 16439 webserver.cc:460] Webserver started at http://127.16.13.195:37871/ using document root <none> and password file <none>
I20240501 14:08:52.717914 17007 ts_manager.cc:194] Registered new tserver with Master: a4a6589543cb487b81dd6f0ec0a85fcb (127.16.13.194:32803)
I20240501 14:08:52.718019 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:52.718066 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:52.718140 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:52.718467 17007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60020
I20240501 14:08:52.718498 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714572529860089-16439-0/minicluster-data/ts-2-root/instance:
uuid: "0333d5b1f6714097804614f828586e44"
format_stamp: "Formatted at 2024-05-01 14:08:52 on dist-test-slave-mgxc"
I20240501 14:08:52.719862 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:52.720757 17223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:52.721045 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:52.721109 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714572529860089-16439-0/minicluster-data/ts-2-root
uuid: "0333d5b1f6714097804614f828586e44"
format_stamp: "Formatted at 2024-05-01 14:08:52 on dist-test-slave-mgxc"
I20240501 14:08:52.721172 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714572529860089-16439-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714572529860089-16439-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714572529860089-16439-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:52.725069 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:52.725301 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:52.725505 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:52.725876 16439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:52.725914 16439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:52.725955 16439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:52.725986 16439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:52.730000 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.195:35511
I20240501 14:08:52.730093 17286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.195:35511 every 8 connection(s)
I20240501 14:08:52.734189 17287 heartbeater.cc:347] Connected to a master server at 127.16.13.254:42089
I20240501 14:08:52.734280 17287 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:52.734449 17287 heartbeater.cc:511] Master 127.16.13.254:42089 requested a full tablet report, sending...
I20240501 14:08:52.734982 17007 ts_manager.cc:194] Registered new tserver with Master: 0333d5b1f6714097804614f828586e44 (127.16.13.195:35511)
I20240501 14:08:52.735466 17007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60028
I20240501 14:08:52.735960 16439 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005582702s
I20240501 14:08:53.695792 17136 heartbeater.cc:503] Master 127.16.13.254:42089 was elected leader, sending a full tablet report...
I20240501 14:08:53.719303 17212 heartbeater.cc:503] Master 127.16.13.254:42089 was elected leader, sending a full tablet report...
I20240501 14:08:53.736375 17287 heartbeater.cc:503] Master 127.16.13.254:42089 was elected leader, sending a full tablet report...
I20240501 14:08:53.761976 16439 test_util.cc:268] Using random seed: 114362292
I20240501 14:08:53.770444 17007 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60042:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 14:08:53.771011 17007 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 14:08:53.783811 17251 tablet_service.cc:1467] Processing CreateTablet for tablet 86983f2828e143df8ef2aa8f61b46353 (DEFAULT_TABLE table=test-workload [id=231ae56594b44ec0a0a73ab662495a31]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240501 14:08:53.784183 17176 tablet_service.cc:1467] Processing CreateTablet for tablet 86983f2828e143df8ef2aa8f61b46353 (DEFAULT_TABLE table=test-workload [id=231ae56594b44ec0a0a73ab662495a31]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240501 14:08:53.784230 17251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86983f2828e143df8ef2aa8f61b46353. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.784207 17249 tablet_service.cc:1467] Processing CreateTablet for tablet e9b3a1494abb4116ae44a9711c535435 (DEFAULT_TABLE table=test-workload [id=231ae56594b44ec0a0a73ab662495a31]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240501 14:08:53.784447 17175 tablet_service.cc:1467] Processing CreateTablet for tablet f74eba74eed44278827bd37383ea632b (DEFAULT_TABLE table=test-workload [id=231ae56594b44ec0a0a73ab662495a31]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240501 14:08:53.784526 17249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9b3a1494abb4116ae44a9711c535435. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.784648 17245 tablet_service.cc:1467] Processing CreateTablet for tablet 08d8843520d54b7fb49be8b2cdead7b7 (DEFAULT_TABLE table=test-workload [id=231ae56594b44ec0a0a73ab662495a31]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240501 14:08:53.784864 17174 tablet_service.cc:1467] Processing CreateTablet for tablet e9b3a1494abb4116ae44a9711c535435 (DEFAULT_TABLE table=test-workload [id=231ae56594b44ec0a0a73ab662495a31]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240501 14:08:53.785259 17173 tablet_service.cc:1467] Processing CreateTablet for tablet b6e8031453c845e083866e9706373eb8 (DEFAULT_TABLE table=test-workload [id=231ae56594b44ec0a0a73ab662495a31]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240501 14:08:53.785683 17172 tablet_service.cc:1467] Processing CreateTablet for tablet eb2837529c914f5192cdaf3fae16afc1 (DEFAULT_TABLE table=test-workload [id=231ae56594b44ec0a0a73ab662495a31]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240501 14:08:53.786077 17171 tablet_service.cc:1467] Processing CreateTablet for tablet 11957e1d5b084712b057d53625ec5e23 (DEFAULT_TABLE table=test-workload [id=231ae56594b44ec0a0a73ab662495a31]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240501 14:08:53.786449 17170 tablet_service.cc:1467] Processing CreateTablet for tablet 08d8843520d54b7fb49be8b2cdead7b7 (DEFAULT_TABLE table=test-workload [id=231ae56594b44ec0a0a73ab662495a31]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240501 14:08:53.786957 17169 tablet_service.cc:1467] Processing CreateTablet for tablet 370d7c1dc64d40fa87489f566515bf30 (DEFAULT_TABLE table=test-workload [id=231ae56594b44ec0a0a73ab662495a31]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240501 14:08:53.784865 17245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08d8843520d54b7fb49be8b2cdead7b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.787218 17248 tablet_service.cc:1467] Processing CreateTablet for tablet b6e8031453c845e083866e9706373eb8 (DEFAULT_TABLE table=test-workload [id=231ae56594b44ec0a0a73ab662495a31]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240501 14:08:53.784431 17176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86983f2828e143df8ef2aa8f61b46353. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.787508 17248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6e8031453c845e083866e9706373eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.787640 17172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb2837529c914f5192cdaf3fae16afc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.787791 17169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 370d7c1dc64d40fa87489f566515bf30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.788312 17174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9b3a1494abb4116ae44a9711c535435. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.788271 17247 tablet_service.cc:1467] Processing CreateTablet for tablet eb2837529c914f5192cdaf3fae16afc1 (DEFAULT_TABLE table=test-workload [id=231ae56594b44ec0a0a73ab662495a31]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240501 14:08:53.788499 17247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb2837529c914f5192cdaf3fae16afc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.788904 17170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08d8843520d54b7fb49be8b2cdead7b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.790309 17173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6e8031453c845e083866e9706373eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.790429 17246 tablet_service.cc:1467] Processing CreateTablet for tablet 11957e1d5b084712b057d53625ec5e23 (DEFAULT_TABLE table=test-workload [id=231ae56594b44ec0a0a73ab662495a31]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240501 14:08:53.790653 17246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11957e1d5b084712b057d53625ec5e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.790836 17171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11957e1d5b084712b057d53625ec5e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.792053 17175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f74eba74eed44278827bd37383ea632b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.792213 17244 tablet_service.cc:1467] Processing CreateTablet for tablet 370d7c1dc64d40fa87489f566515bf30 (DEFAULT_TABLE table=test-workload [id=231ae56594b44ec0a0a73ab662495a31]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240501 14:08:53.792439 17244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 370d7c1dc64d40fa87489f566515bf30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.793782 17307 tablet_bootstrap.cc:492] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44: Bootstrap starting.
I20240501 14:08:53.794615 17250 tablet_service.cc:1467] Processing CreateTablet for tablet f74eba74eed44278827bd37383ea632b (DEFAULT_TABLE table=test-workload [id=231ae56594b44ec0a0a73ab662495a31]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240501 14:08:53.794826 17250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f74eba74eed44278827bd37383ea632b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.794823 17307 tablet_bootstrap.cc:654] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.794991 17308 tablet_bootstrap.cc:492] T 370d7c1dc64d40fa87489f566515bf30 P a4a6589543cb487b81dd6f0ec0a85fcb: Bootstrap starting.
I20240501 14:08:53.795974 17308 tablet_bootstrap.cc:654] T 370d7c1dc64d40fa87489f566515bf30 P a4a6589543cb487b81dd6f0ec0a85fcb: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.796124 17307 tablet_bootstrap.cc:492] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44: No bootstrap required, opened a new log
I20240501 14:08:53.796213 17307 ts_tablet_manager.cc:1367] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:53.796686 17307 raft_consensus.cc:348] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.796797 17307 raft_consensus.cc:374] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.796826 17307 raft_consensus.cc:729] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0333d5b1f6714097804614f828586e44, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.796979 17307 consensus_queue.cc:260] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.797070 17308 tablet_bootstrap.cc:492] T 370d7c1dc64d40fa87489f566515bf30 P a4a6589543cb487b81dd6f0ec0a85fcb: No bootstrap required, opened a new log
I20240501 14:08:53.797144 17308 ts_tablet_manager.cc:1367] T 370d7c1dc64d40fa87489f566515bf30 P a4a6589543cb487b81dd6f0ec0a85fcb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:08:53.797351 17307 ts_tablet_manager.cc:1398] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:53.797617 17308 raft_consensus.cc:348] T 370d7c1dc64d40fa87489f566515bf30 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.797724 17308 raft_consensus.cc:374] T 370d7c1dc64d40fa87489f566515bf30 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.797760 17308 raft_consensus.cc:729] T 370d7c1dc64d40fa87489f566515bf30 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4a6589543cb487b81dd6f0ec0a85fcb, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.797888 17308 consensus_queue.cc:260] T 370d7c1dc64d40fa87489f566515bf30 P a4a6589543cb487b81dd6f0ec0a85fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.798101 17307 tablet_bootstrap.cc:492] T 86983f2828e143df8ef2aa8f61b46353 P 0333d5b1f6714097804614f828586e44: Bootstrap starting.
I20240501 14:08:53.798214 17308 ts_tablet_manager.cc:1398] T 370d7c1dc64d40fa87489f566515bf30 P a4a6589543cb487b81dd6f0ec0a85fcb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:08:53.798389 17308 tablet_bootstrap.cc:492] T e9b3a1494abb4116ae44a9711c535435 P a4a6589543cb487b81dd6f0ec0a85fcb: Bootstrap starting.
I20240501 14:08:53.798981 17307 tablet_bootstrap.cc:654] T 86983f2828e143df8ef2aa8f61b46353 P 0333d5b1f6714097804614f828586e44: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.799319 17308 tablet_bootstrap.cc:654] T e9b3a1494abb4116ae44a9711c535435 P a4a6589543cb487b81dd6f0ec0a85fcb: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.800562 17308 tablet_bootstrap.cc:492] T e9b3a1494abb4116ae44a9711c535435 P a4a6589543cb487b81dd6f0ec0a85fcb: No bootstrap required, opened a new log
I20240501 14:08:53.800648 17308 ts_tablet_manager.cc:1367] T e9b3a1494abb4116ae44a9711c535435 P a4a6589543cb487b81dd6f0ec0a85fcb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:08:53.801038 17308 raft_consensus.cc:348] T e9b3a1494abb4116ae44a9711c535435 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.801147 17308 raft_consensus.cc:374] T e9b3a1494abb4116ae44a9711c535435 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.801177 17308 raft_consensus.cc:729] T e9b3a1494abb4116ae44a9711c535435 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4a6589543cb487b81dd6f0ec0a85fcb, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.801290 17308 consensus_queue.cc:260] T e9b3a1494abb4116ae44a9711c535435 P a4a6589543cb487b81dd6f0ec0a85fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.801513 17308 ts_tablet_manager.cc:1398] T e9b3a1494abb4116ae44a9711c535435 P a4a6589543cb487b81dd6f0ec0a85fcb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:08:53.801666 17308 tablet_bootstrap.cc:492] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb: Bootstrap starting.
I20240501 14:08:53.802495 17308 tablet_bootstrap.cc:654] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.804090 17308 tablet_bootstrap.cc:492] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb: No bootstrap required, opened a new log
I20240501 14:08:53.804251 17308 ts_tablet_manager.cc:1367] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:08:53.804365 17307 tablet_bootstrap.cc:492] T 86983f2828e143df8ef2aa8f61b46353 P 0333d5b1f6714097804614f828586e44: No bootstrap required, opened a new log
I20240501 14:08:53.804469 17307 ts_tablet_manager.cc:1367] T 86983f2828e143df8ef2aa8f61b46353 P 0333d5b1f6714097804614f828586e44: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:08:53.804586 17094 tablet_service.cc:1467] Processing CreateTablet for tablet 08d8843520d54b7fb49be8b2cdead7b7 (DEFAULT_TABLE table=test-workload [id=231ae56594b44ec0a0a73ab662495a31]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240501 14:08:53.804697 17099 tablet_service.cc:1467] Processing CreateTablet for tablet f74eba74eed44278827bd37383ea632b (DEFAULT_TABLE table=test-workload [id=231ae56594b44ec0a0a73ab662495a31]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240501 14:08:53.804832 17094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08d8843520d54b7fb49be8b2cdead7b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.804885 17307 raft_consensus.cc:348] T 86983f2828e143df8ef2aa8f61b46353 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } }
I20240501 14:08:53.804239 17100 tablet_service.cc:1467] Processing CreateTablet for tablet 86983f2828e143df8ef2aa8f61b46353 (DEFAULT_TABLE table=test-workload [id=231ae56594b44ec0a0a73ab662495a31]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240501 14:08:53.805006 17308 raft_consensus.cc:348] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.805027 17307 raft_consensus.cc:374] T 86983f2828e143df8ef2aa8f61b46353 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.805305 17307 raft_consensus.cc:729] T 86983f2828e143df8ef2aa8f61b46353 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0333d5b1f6714097804614f828586e44, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.805214 17100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86983f2828e143df8ef2aa8f61b46353. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.805444 17097 tablet_service.cc:1467] Processing CreateTablet for tablet b6e8031453c845e083866e9706373eb8 (DEFAULT_TABLE table=test-workload [id=231ae56594b44ec0a0a73ab662495a31]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240501 14:08:53.805719 17097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6e8031453c845e083866e9706373eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.805265 17308 raft_consensus.cc:374] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.807256 17308 raft_consensus.cc:729] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4a6589543cb487b81dd6f0ec0a85fcb, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.807389 17308 consensus_queue.cc:260] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.807644 17096 tablet_service.cc:1467] Processing CreateTablet for tablet eb2837529c914f5192cdaf3fae16afc1 (DEFAULT_TABLE table=test-workload [id=231ae56594b44ec0a0a73ab662495a31]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240501 14:08:53.807827 17307 consensus_queue.cc:260] T 86983f2828e143df8ef2aa8f61b46353 P 0333d5b1f6714097804614f828586e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } }
I20240501 14:08:53.807894 17096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb2837529c914f5192cdaf3fae16afc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.808130 17307 ts_tablet_manager.cc:1398] T 86983f2828e143df8ef2aa8f61b46353 P 0333d5b1f6714097804614f828586e44: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 14:08:53.808344 17307 tablet_bootstrap.cc:492] T eb2837529c914f5192cdaf3fae16afc1 P 0333d5b1f6714097804614f828586e44: Bootstrap starting.
I20240501 14:08:53.809408 17307 tablet_bootstrap.cc:654] T eb2837529c914f5192cdaf3fae16afc1 P 0333d5b1f6714097804614f828586e44: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.809422 17093 tablet_service.cc:1467] Processing CreateTablet for tablet 370d7c1dc64d40fa87489f566515bf30 (DEFAULT_TABLE table=test-workload [id=231ae56594b44ec0a0a73ab662495a31]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240501 14:08:53.809693 17093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 370d7c1dc64d40fa87489f566515bf30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.809762 17095 tablet_service.cc:1467] Processing CreateTablet for tablet 11957e1d5b084712b057d53625ec5e23 (DEFAULT_TABLE table=test-workload [id=231ae56594b44ec0a0a73ab662495a31]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240501 14:08:53.810035 17095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11957e1d5b084712b057d53625ec5e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.811728 17307 tablet_bootstrap.cc:492] T eb2837529c914f5192cdaf3fae16afc1 P 0333d5b1f6714097804614f828586e44: No bootstrap required, opened a new log
I20240501 14:08:53.811821 17307 ts_tablet_manager.cc:1367] T eb2837529c914f5192cdaf3fae16afc1 P 0333d5b1f6714097804614f828586e44: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:08:53.805157 17098 tablet_service.cc:1467] Processing CreateTablet for tablet e9b3a1494abb4116ae44a9711c535435 (DEFAULT_TABLE table=test-workload [id=231ae56594b44ec0a0a73ab662495a31]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240501 14:08:53.812332 17307 raft_consensus.cc:348] T eb2837529c914f5192cdaf3fae16afc1 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.812420 17098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9b3a1494abb4116ae44a9711c535435. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.812438 17307 raft_consensus.cc:374] T eb2837529c914f5192cdaf3fae16afc1 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.812477 17307 raft_consensus.cc:729] T eb2837529c914f5192cdaf3fae16afc1 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0333d5b1f6714097804614f828586e44, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.812592 17307 consensus_queue.cc:260] T eb2837529c914f5192cdaf3fae16afc1 P 0333d5b1f6714097804614f828586e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.813216 17307 ts_tablet_manager.cc:1398] T eb2837529c914f5192cdaf3fae16afc1 P 0333d5b1f6714097804614f828586e44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:53.813396 17307 tablet_bootstrap.cc:492] T 08d8843520d54b7fb49be8b2cdead7b7 P 0333d5b1f6714097804614f828586e44: Bootstrap starting.
I20240501 14:08:53.814318 17099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f74eba74eed44278827bd37383ea632b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.807703 17308 ts_tablet_manager.cc:1398] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:08:53.815143 17308 tablet_bootstrap.cc:492] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb: Bootstrap starting.
I20240501 14:08:53.816057 17308 tablet_bootstrap.cc:654] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.816112 17316 tablet_bootstrap.cc:492] T 08d8843520d54b7fb49be8b2cdead7b7 P 694d2b1e07b84029a6232d1bbfba3be6: Bootstrap starting.
I20240501 14:08:53.814308 17307 tablet_bootstrap.cc:654] T 08d8843520d54b7fb49be8b2cdead7b7 P 0333d5b1f6714097804614f828586e44: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.817119 17316 tablet_bootstrap.cc:654] T 08d8843520d54b7fb49be8b2cdead7b7 P 694d2b1e07b84029a6232d1bbfba3be6: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.817819 17307 tablet_bootstrap.cc:492] T 08d8843520d54b7fb49be8b2cdead7b7 P 0333d5b1f6714097804614f828586e44: No bootstrap required, opened a new log
I20240501 14:08:53.817924 17307 ts_tablet_manager.cc:1367] T 08d8843520d54b7fb49be8b2cdead7b7 P 0333d5b1f6714097804614f828586e44: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240501 14:08:53.817997 17308 tablet_bootstrap.cc:492] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb: No bootstrap required, opened a new log
I20240501 14:08:53.818086 17308 ts_tablet_manager.cc:1367] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:08:53.818243 17316 tablet_bootstrap.cc:492] T 08d8843520d54b7fb49be8b2cdead7b7 P 694d2b1e07b84029a6232d1bbfba3be6: No bootstrap required, opened a new log
I20240501 14:08:53.818321 17316 ts_tablet_manager.cc:1367] T 08d8843520d54b7fb49be8b2cdead7b7 P 694d2b1e07b84029a6232d1bbfba3be6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:53.818360 17307 raft_consensus.cc:348] T 08d8843520d54b7fb49be8b2cdead7b7 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.818490 17307 raft_consensus.cc:374] T 08d8843520d54b7fb49be8b2cdead7b7 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.818542 17307 raft_consensus.cc:729] T 08d8843520d54b7fb49be8b2cdead7b7 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0333d5b1f6714097804614f828586e44, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.818497 17308 raft_consensus.cc:348] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.818627 17308 raft_consensus.cc:374] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.818668 17308 raft_consensus.cc:729] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4a6589543cb487b81dd6f0ec0a85fcb, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.818795 17308 consensus_queue.cc:260] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.818672 17307 consensus_queue.cc:260] T 08d8843520d54b7fb49be8b2cdead7b7 P 0333d5b1f6714097804614f828586e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.818852 17316 raft_consensus.cc:348] T 08d8843520d54b7fb49be8b2cdead7b7 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.819061 17316 raft_consensus.cc:374] T 08d8843520d54b7fb49be8b2cdead7b7 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.819089 17308 ts_tablet_manager.cc:1398] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:53.819098 17316 raft_consensus.cc:729] T 08d8843520d54b7fb49be8b2cdead7b7 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 694d2b1e07b84029a6232d1bbfba3be6, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.819149 17307 ts_tablet_manager.cc:1398] T 08d8843520d54b7fb49be8b2cdead7b7 P 0333d5b1f6714097804614f828586e44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:53.819285 17308 tablet_bootstrap.cc:492] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb: Bootstrap starting.
I20240501 14:08:53.819766 17316 consensus_queue.cc:260] T 08d8843520d54b7fb49be8b2cdead7b7 P 694d2b1e07b84029a6232d1bbfba3be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.820237 17316 ts_tablet_manager.cc:1398] T 08d8843520d54b7fb49be8b2cdead7b7 P 694d2b1e07b84029a6232d1bbfba3be6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:53.819298 17307 tablet_bootstrap.cc:492] T 370d7c1dc64d40fa87489f566515bf30 P 0333d5b1f6714097804614f828586e44: Bootstrap starting.
I20240501 14:08:53.820511 17316 tablet_bootstrap.cc:492] T 86983f2828e143df8ef2aa8f61b46353 P 694d2b1e07b84029a6232d1bbfba3be6: Bootstrap starting.
I20240501 14:08:53.821251 17307 tablet_bootstrap.cc:654] T 370d7c1dc64d40fa87489f566515bf30 P 0333d5b1f6714097804614f828586e44: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.821606 17316 tablet_bootstrap.cc:654] T 86983f2828e143df8ef2aa8f61b46353 P 694d2b1e07b84029a6232d1bbfba3be6: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.820271 17308 tablet_bootstrap.cc:654] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.822687 17307 tablet_bootstrap.cc:492] T 370d7c1dc64d40fa87489f566515bf30 P 0333d5b1f6714097804614f828586e44: No bootstrap required, opened a new log
I20240501 14:08:53.822770 17307 ts_tablet_manager.cc:1367] T 370d7c1dc64d40fa87489f566515bf30 P 0333d5b1f6714097804614f828586e44: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 14:08:53.822906 17308 tablet_bootstrap.cc:492] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb: No bootstrap required, opened a new log
I20240501 14:08:53.822989 17308 ts_tablet_manager.cc:1367] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:08:53.823181 17307 raft_consensus.cc:348] T 370d7c1dc64d40fa87489f566515bf30 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.823282 17307 raft_consensus.cc:374] T 370d7c1dc64d40fa87489f566515bf30 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.823314 17307 raft_consensus.cc:729] T 370d7c1dc64d40fa87489f566515bf30 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0333d5b1f6714097804614f828586e44, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.823419 17308 raft_consensus.cc:348] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.823505 17308 raft_consensus.cc:374] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.823457 17307 consensus_queue.cc:260] T 370d7c1dc64d40fa87489f566515bf30 P 0333d5b1f6714097804614f828586e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.823541 17316 tablet_bootstrap.cc:492] T 86983f2828e143df8ef2aa8f61b46353 P 694d2b1e07b84029a6232d1bbfba3be6: No bootstrap required, opened a new log
I20240501 14:08:53.823626 17316 ts_tablet_manager.cc:1367] T 86983f2828e143df8ef2aa8f61b46353 P 694d2b1e07b84029a6232d1bbfba3be6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:08:53.823776 17307 ts_tablet_manager.cc:1398] T 370d7c1dc64d40fa87489f566515bf30 P 0333d5b1f6714097804614f828586e44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:53.823536 17308 raft_consensus.cc:729] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4a6589543cb487b81dd6f0ec0a85fcb, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.824241 17308 consensus_queue.cc:260] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.824303 17307 tablet_bootstrap.cc:492] T f74eba74eed44278827bd37383ea632b P 0333d5b1f6714097804614f828586e44: Bootstrap starting.
I20240501 14:08:53.824031 17316 raft_consensus.cc:348] T 86983f2828e143df8ef2aa8f61b46353 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } }
I20240501 14:08:53.824898 17316 raft_consensus.cc:374] T 86983f2828e143df8ef2aa8f61b46353 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.824951 17316 raft_consensus.cc:729] T 86983f2828e143df8ef2aa8f61b46353 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 694d2b1e07b84029a6232d1bbfba3be6, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.825027 17308 ts_tablet_manager.cc:1398] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:53.825073 17316 consensus_queue.cc:260] T 86983f2828e143df8ef2aa8f61b46353 P 694d2b1e07b84029a6232d1bbfba3be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } }
I20240501 14:08:53.825224 17308 tablet_bootstrap.cc:492] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb: Bootstrap starting.
I20240501 14:08:53.825315 17316 ts_tablet_manager.cc:1398] T 86983f2828e143df8ef2aa8f61b46353 P 694d2b1e07b84029a6232d1bbfba3be6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:53.825465 17316 tablet_bootstrap.cc:492] T b6e8031453c845e083866e9706373eb8 P 694d2b1e07b84029a6232d1bbfba3be6: Bootstrap starting.
I20240501 14:08:53.826197 17308 tablet_bootstrap.cc:654] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.826570 17307 tablet_bootstrap.cc:654] T f74eba74eed44278827bd37383ea632b P 0333d5b1f6714097804614f828586e44: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.826246 17316 tablet_bootstrap.cc:654] T b6e8031453c845e083866e9706373eb8 P 694d2b1e07b84029a6232d1bbfba3be6: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.827658 17307 tablet_bootstrap.cc:492] T f74eba74eed44278827bd37383ea632b P 0333d5b1f6714097804614f828586e44: No bootstrap required, opened a new log
I20240501 14:08:53.827699 17316 tablet_bootstrap.cc:492] T b6e8031453c845e083866e9706373eb8 P 694d2b1e07b84029a6232d1bbfba3be6: No bootstrap required, opened a new log
I20240501 14:08:53.827754 17307 ts_tablet_manager.cc:1367] T f74eba74eed44278827bd37383ea632b P 0333d5b1f6714097804614f828586e44: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:08:53.827793 17316 ts_tablet_manager.cc:1367] T b6e8031453c845e083866e9706373eb8 P 694d2b1e07b84029a6232d1bbfba3be6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:53.828187 17308 tablet_bootstrap.cc:492] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb: No bootstrap required, opened a new log
I20240501 14:08:53.828298 17308 ts_tablet_manager.cc:1367] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:08:53.828168 17307 raft_consensus.cc:348] T f74eba74eed44278827bd37383ea632b P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.828366 17307 raft_consensus.cc:374] T f74eba74eed44278827bd37383ea632b P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.828414 17307 raft_consensus.cc:729] T f74eba74eed44278827bd37383ea632b P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0333d5b1f6714097804614f828586e44, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.828526 17307 consensus_queue.cc:260] T f74eba74eed44278827bd37383ea632b P 0333d5b1f6714097804614f828586e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.828670 17308 raft_consensus.cc:348] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } }
I20240501 14:08:53.828758 17308 raft_consensus.cc:374] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.828269 17316 raft_consensus.cc:348] T b6e8031453c845e083866e9706373eb8 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.828801 17308 raft_consensus.cc:729] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4a6589543cb487b81dd6f0ec0a85fcb, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.828840 17316 raft_consensus.cc:374] T b6e8031453c845e083866e9706373eb8 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.828886 17316 raft_consensus.cc:729] T b6e8031453c845e083866e9706373eb8 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 694d2b1e07b84029a6232d1bbfba3be6, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.828915 17308 consensus_queue.cc:260] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } }
I20240501 14:08:53.828994 17316 consensus_queue.cc:260] T b6e8031453c845e083866e9706373eb8 P 694d2b1e07b84029a6232d1bbfba3be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.829160 17308 ts_tablet_manager.cc:1398] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:53.829269 17316 ts_tablet_manager.cc:1398] T b6e8031453c845e083866e9706373eb8 P 694d2b1e07b84029a6232d1bbfba3be6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:08:53.829305 17308 tablet_bootstrap.cc:492] T f74eba74eed44278827bd37383ea632b P a4a6589543cb487b81dd6f0ec0a85fcb: Bootstrap starting.
I20240501 14:08:53.829420 17316 tablet_bootstrap.cc:492] T eb2837529c914f5192cdaf3fae16afc1 P 694d2b1e07b84029a6232d1bbfba3be6: Bootstrap starting.
I20240501 14:08:53.830148 17308 tablet_bootstrap.cc:654] T f74eba74eed44278827bd37383ea632b P a4a6589543cb487b81dd6f0ec0a85fcb: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.830344 17316 tablet_bootstrap.cc:654] T eb2837529c914f5192cdaf3fae16afc1 P 694d2b1e07b84029a6232d1bbfba3be6: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.828806 17307 ts_tablet_manager.cc:1398] T f74eba74eed44278827bd37383ea632b P 0333d5b1f6714097804614f828586e44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:53.831113 17308 tablet_bootstrap.cc:492] T f74eba74eed44278827bd37383ea632b P a4a6589543cb487b81dd6f0ec0a85fcb: No bootstrap required, opened a new log
I20240501 14:08:53.831137 17307 tablet_bootstrap.cc:492] T 11957e1d5b084712b057d53625ec5e23 P 0333d5b1f6714097804614f828586e44: Bootstrap starting.
I20240501 14:08:53.831202 17308 ts_tablet_manager.cc:1367] T f74eba74eed44278827bd37383ea632b P a4a6589543cb487b81dd6f0ec0a85fcb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:53.831583 17308 raft_consensus.cc:348] T f74eba74eed44278827bd37383ea632b P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.831686 17308 raft_consensus.cc:374] T f74eba74eed44278827bd37383ea632b P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.831715 17308 raft_consensus.cc:729] T f74eba74eed44278827bd37383ea632b P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4a6589543cb487b81dd6f0ec0a85fcb, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.832046 17307 tablet_bootstrap.cc:654] T 11957e1d5b084712b057d53625ec5e23 P 0333d5b1f6714097804614f828586e44: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.832391 17308 consensus_queue.cc:260] T f74eba74eed44278827bd37383ea632b P a4a6589543cb487b81dd6f0ec0a85fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.832720 17316 tablet_bootstrap.cc:492] T eb2837529c914f5192cdaf3fae16afc1 P 694d2b1e07b84029a6232d1bbfba3be6: No bootstrap required, opened a new log
I20240501 14:08:53.832813 17316 ts_tablet_manager.cc:1367] T eb2837529c914f5192cdaf3fae16afc1 P 694d2b1e07b84029a6232d1bbfba3be6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:08:53.833027 17308 ts_tablet_manager.cc:1398] T f74eba74eed44278827bd37383ea632b P a4a6589543cb487b81dd6f0ec0a85fcb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:53.833209 17308 tablet_bootstrap.cc:492] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb: Bootstrap starting.
I20240501 14:08:53.833202 17316 raft_consensus.cc:348] T eb2837529c914f5192cdaf3fae16afc1 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.833323 17316 raft_consensus.cc:374] T eb2837529c914f5192cdaf3fae16afc1 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.833361 17316 raft_consensus.cc:729] T eb2837529c914f5192cdaf3fae16afc1 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 694d2b1e07b84029a6232d1bbfba3be6, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.833405 17307 tablet_bootstrap.cc:492] T 11957e1d5b084712b057d53625ec5e23 P 0333d5b1f6714097804614f828586e44: No bootstrap required, opened a new log
I20240501 14:08:53.833492 17307 ts_tablet_manager.cc:1367] T 11957e1d5b084712b057d53625ec5e23 P 0333d5b1f6714097804614f828586e44: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:08:53.833729 17316 consensus_queue.cc:260] T eb2837529c914f5192cdaf3fae16afc1 P 694d2b1e07b84029a6232d1bbfba3be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.834106 17308 tablet_bootstrap.cc:654] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.834743 17307 raft_consensus.cc:348] T 11957e1d5b084712b057d53625ec5e23 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.834820 17316 ts_tablet_manager.cc:1398] T eb2837529c914f5192cdaf3fae16afc1 P 694d2b1e07b84029a6232d1bbfba3be6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:08:53.834998 17316 tablet_bootstrap.cc:492] T 370d7c1dc64d40fa87489f566515bf30 P 694d2b1e07b84029a6232d1bbfba3be6: Bootstrap starting.
I20240501 14:08:53.835053 17308 tablet_bootstrap.cc:492] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb: No bootstrap required, opened a new log
I20240501 14:08:53.835136 17308 ts_tablet_manager.cc:1367] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:53.835541 17308 raft_consensus.cc:348] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.835647 17308 raft_consensus.cc:374] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.835686 17308 raft_consensus.cc:729] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4a6589543cb487b81dd6f0ec0a85fcb, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.835811 17308 consensus_queue.cc:260] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.835867 17316 tablet_bootstrap.cc:654] T 370d7c1dc64d40fa87489f566515bf30 P 694d2b1e07b84029a6232d1bbfba3be6: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.835899 17312 raft_consensus.cc:482] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:53.835985 17312 raft_consensus.cc:504] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.836361 17308 ts_tablet_manager.cc:1398] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:53.836547 17312 leader_election.cc:288] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 694d2b1e07b84029a6232d1bbfba3be6 (127.16.13.193:33287), 0333d5b1f6714097804614f828586e44 (127.16.13.195:35511)
I20240501 14:08:53.834833 17307 raft_consensus.cc:374] T 11957e1d5b084712b057d53625ec5e23 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.836828 17307 raft_consensus.cc:729] T 11957e1d5b084712b057d53625ec5e23 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0333d5b1f6714097804614f828586e44, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.836959 17307 consensus_queue.cc:260] T 11957e1d5b084712b057d53625ec5e23 P 0333d5b1f6714097804614f828586e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.837204 17307 ts_tablet_manager.cc:1398] T 11957e1d5b084712b057d53625ec5e23 P 0333d5b1f6714097804614f828586e44: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:08:53.837357 17307 tablet_bootstrap.cc:492] T b6e8031453c845e083866e9706373eb8 P 0333d5b1f6714097804614f828586e44: Bootstrap starting.
I20240501 14:08:53.838255 17307 tablet_bootstrap.cc:654] T b6e8031453c845e083866e9706373eb8 P 0333d5b1f6714097804614f828586e44: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.838594 17316 tablet_bootstrap.cc:492] T 370d7c1dc64d40fa87489f566515bf30 P 694d2b1e07b84029a6232d1bbfba3be6: No bootstrap required, opened a new log
I20240501 14:08:53.838673 17316 ts_tablet_manager.cc:1367] T 370d7c1dc64d40fa87489f566515bf30 P 694d2b1e07b84029a6232d1bbfba3be6: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:08:53.839078 17316 raft_consensus.cc:348] T 370d7c1dc64d40fa87489f566515bf30 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.839238 17316 raft_consensus.cc:374] T 370d7c1dc64d40fa87489f566515bf30 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.839274 17316 raft_consensus.cc:729] T 370d7c1dc64d40fa87489f566515bf30 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 694d2b1e07b84029a6232d1bbfba3be6, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.839375 17316 consensus_queue.cc:260] T 370d7c1dc64d40fa87489f566515bf30 P 694d2b1e07b84029a6232d1bbfba3be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.839500 17307 tablet_bootstrap.cc:492] T b6e8031453c845e083866e9706373eb8 P 0333d5b1f6714097804614f828586e44: No bootstrap required, opened a new log
I20240501 14:08:53.839563 17307 ts_tablet_manager.cc:1367] T b6e8031453c845e083866e9706373eb8 P 0333d5b1f6714097804614f828586e44: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:08:53.839609 17316 ts_tablet_manager.cc:1398] T 370d7c1dc64d40fa87489f566515bf30 P 694d2b1e07b84029a6232d1bbfba3be6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:53.839774 17316 tablet_bootstrap.cc:492] T e9b3a1494abb4116ae44a9711c535435 P 694d2b1e07b84029a6232d1bbfba3be6: Bootstrap starting.
I20240501 14:08:53.840327 17307 raft_consensus.cc:348] T b6e8031453c845e083866e9706373eb8 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.840409 17307 raft_consensus.cc:374] T b6e8031453c845e083866e9706373eb8 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.840462 17307 raft_consensus.cc:729] T b6e8031453c845e083866e9706373eb8 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0333d5b1f6714097804614f828586e44, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.840577 17307 consensus_queue.cc:260] T b6e8031453c845e083866e9706373eb8 P 0333d5b1f6714097804614f828586e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.840665 17316 tablet_bootstrap.cc:654] T e9b3a1494abb4116ae44a9711c535435 P 694d2b1e07b84029a6232d1bbfba3be6: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.840863 17307 ts_tablet_manager.cc:1398] T b6e8031453c845e083866e9706373eb8 P 0333d5b1f6714097804614f828586e44: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:08:53.841425 17110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb2837529c914f5192cdaf3fae16afc1" candidate_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "694d2b1e07b84029a6232d1bbfba3be6" is_pre_election: true
I20240501 14:08:53.841555 17110 raft_consensus.cc:2454] T eb2837529c914f5192cdaf3fae16afc1 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4a6589543cb487b81dd6f0ec0a85fcb in term 0.
I20240501 14:08:53.841603 17316 tablet_bootstrap.cc:492] T e9b3a1494abb4116ae44a9711c535435 P 694d2b1e07b84029a6232d1bbfba3be6: No bootstrap required, opened a new log
I20240501 14:08:53.841688 17316 ts_tablet_manager.cc:1367] T e9b3a1494abb4116ae44a9711c535435 P 694d2b1e07b84029a6232d1bbfba3be6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:53.841955 17152 leader_election.cc:302] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 694d2b1e07b84029a6232d1bbfba3be6, a4a6589543cb487b81dd6f0ec0a85fcb; no voters:
I20240501 14:08:53.842077 17316 raft_consensus.cc:348] T e9b3a1494abb4116ae44a9711c535435 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.842159 17316 raft_consensus.cc:374] T e9b3a1494abb4116ae44a9711c535435 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.842226 17316 raft_consensus.cc:729] T e9b3a1494abb4116ae44a9711c535435 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 694d2b1e07b84029a6232d1bbfba3be6, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.842322 17316 consensus_queue.cc:260] T e9b3a1494abb4116ae44a9711c535435 P 694d2b1e07b84029a6232d1bbfba3be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.842571 17316 ts_tablet_manager.cc:1398] T e9b3a1494abb4116ae44a9711c535435 P 694d2b1e07b84029a6232d1bbfba3be6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:53.842720 17316 tablet_bootstrap.cc:492] T 11957e1d5b084712b057d53625ec5e23 P 694d2b1e07b84029a6232d1bbfba3be6: Bootstrap starting.
I20240501 14:08:53.843542 17316 tablet_bootstrap.cc:654] T 11957e1d5b084712b057d53625ec5e23 P 694d2b1e07b84029a6232d1bbfba3be6: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.842149 17332 raft_consensus.cc:2781] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:53.844035 17332 raft_consensus.cc:482] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:53.844092 17332 raft_consensus.cc:3037] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:53.844314 17312 raft_consensus.cc:482] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:53.844398 17312 raft_consensus.cc:504] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.844749 17312 leader_election.cc:288] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 694d2b1e07b84029a6232d1bbfba3be6 (127.16.13.193:33287), 0333d5b1f6714097804614f828586e44 (127.16.13.195:35511)
I20240501 14:08:53.844969 17110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11957e1d5b084712b057d53625ec5e23" candidate_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "694d2b1e07b84029a6232d1bbfba3be6" is_pre_election: true
W20240501 14:08:53.845392 17152 leader_election.cc:341] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 694d2b1e07b84029a6232d1bbfba3be6 (127.16.13.193:33287): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:08:53.845628 17316 tablet_bootstrap.cc:492] T 11957e1d5b084712b057d53625ec5e23 P 694d2b1e07b84029a6232d1bbfba3be6: No bootstrap required, opened a new log
I20240501 14:08:53.845633 17261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb2837529c914f5192cdaf3fae16afc1" candidate_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0333d5b1f6714097804614f828586e44" is_pre_election: true
I20240501 14:08:53.845716 17316 ts_tablet_manager.cc:1367] T 11957e1d5b084712b057d53625ec5e23 P 694d2b1e07b84029a6232d1bbfba3be6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:08:53.845680 17260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11957e1d5b084712b057d53625ec5e23" candidate_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0333d5b1f6714097804614f828586e44" is_pre_election: true
I20240501 14:08:53.845783 17261 raft_consensus.cc:2454] T eb2837529c914f5192cdaf3fae16afc1 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4a6589543cb487b81dd6f0ec0a85fcb in term 0.
I20240501 14:08:53.845829 17260 raft_consensus.cc:2454] T 11957e1d5b084712b057d53625ec5e23 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4a6589543cb487b81dd6f0ec0a85fcb in term 0.
I20240501 14:08:53.846103 17151 leader_election.cc:302] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0333d5b1f6714097804614f828586e44, a4a6589543cb487b81dd6f0ec0a85fcb; no voters: 694d2b1e07b84029a6232d1bbfba3be6
I20240501 14:08:53.846251 17312 raft_consensus.cc:2781] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:53.846251 17316 raft_consensus.cc:348] T 11957e1d5b084712b057d53625ec5e23 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.846355 17312 raft_consensus.cc:482] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:53.846369 17316 raft_consensus.cc:374] T 11957e1d5b084712b057d53625ec5e23 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.846390 17312 raft_consensus.cc:3037] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:53.846429 17316 raft_consensus.cc:729] T 11957e1d5b084712b057d53625ec5e23 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 694d2b1e07b84029a6232d1bbfba3be6, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.846544 17316 consensus_queue.cc:260] T 11957e1d5b084712b057d53625ec5e23 P 694d2b1e07b84029a6232d1bbfba3be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.846774 17316 ts_tablet_manager.cc:1398] T 11957e1d5b084712b057d53625ec5e23 P 694d2b1e07b84029a6232d1bbfba3be6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:53.846972 17316 tablet_bootstrap.cc:492] T f74eba74eed44278827bd37383ea632b P 694d2b1e07b84029a6232d1bbfba3be6: Bootstrap starting.
I20240501 14:08:53.847676 17316 tablet_bootstrap.cc:654] T f74eba74eed44278827bd37383ea632b P 694d2b1e07b84029a6232d1bbfba3be6: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.848134 17332 raft_consensus.cc:504] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.848536 17332 leader_election.cc:288] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb [CANDIDATE]: Term 1 election: Requested vote from peers 694d2b1e07b84029a6232d1bbfba3be6 (127.16.13.193:33287), 0333d5b1f6714097804614f828586e44 (127.16.13.195:35511)
I20240501 14:08:53.848742 17110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb2837529c914f5192cdaf3fae16afc1" candidate_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "694d2b1e07b84029a6232d1bbfba3be6"
I20240501 14:08:53.848800 17261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb2837529c914f5192cdaf3fae16afc1" candidate_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0333d5b1f6714097804614f828586e44"
I20240501 14:08:53.848839 17110 raft_consensus.cc:3037] T eb2837529c914f5192cdaf3fae16afc1 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:53.848870 17261 raft_consensus.cc:3037] T eb2837529c914f5192cdaf3fae16afc1 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:53.849745 17312 raft_consensus.cc:504] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.849818 17261 raft_consensus.cc:2454] T eb2837529c914f5192cdaf3fae16afc1 P 0333d5b1f6714097804614f828586e44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4a6589543cb487b81dd6f0ec0a85fcb in term 1.
I20240501 14:08:53.850041 17110 raft_consensus.cc:2454] T eb2837529c914f5192cdaf3fae16afc1 P 694d2b1e07b84029a6232d1bbfba3be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4a6589543cb487b81dd6f0ec0a85fcb in term 1.
I20240501 14:08:53.850100 17151 leader_election.cc:302] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0333d5b1f6714097804614f828586e44, a4a6589543cb487b81dd6f0ec0a85fcb; no voters:
I20240501 14:08:53.850116 17312 leader_election.cc:288] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb [CANDIDATE]: Term 1 election: Requested vote from peers 694d2b1e07b84029a6232d1bbfba3be6 (127.16.13.193:33287), 0333d5b1f6714097804614f828586e44 (127.16.13.195:35511)
I20240501 14:08:53.850495 17261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11957e1d5b084712b057d53625ec5e23" candidate_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0333d5b1f6714097804614f828586e44"
I20240501 14:08:53.850595 17261 raft_consensus.cc:3037] T 11957e1d5b084712b057d53625ec5e23 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:53.850718 17311 raft_consensus.cc:482] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:53.850754 17312 raft_consensus.cc:2781] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:53.850819 17312 raft_consensus.cc:686] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 LEADER]: Becoming Leader. State: Replica: a4a6589543cb487b81dd6f0ec0a85fcb, State: Running, Role: LEADER
I20240501 14:08:53.850790 17311 raft_consensus.cc:504] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.850917 17312 consensus_queue.cc:237] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.851197 17311 leader_election.cc:288] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 694d2b1e07b84029a6232d1bbfba3be6 (127.16.13.193:33287), a4a6589543cb487b81dd6f0ec0a85fcb (127.16.13.194:32803)
I20240501 14:08:53.851464 17316 tablet_bootstrap.cc:492] T f74eba74eed44278827bd37383ea632b P 694d2b1e07b84029a6232d1bbfba3be6: No bootstrap required, opened a new log
I20240501 14:08:53.851542 17316 ts_tablet_manager.cc:1367] T f74eba74eed44278827bd37383ea632b P 694d2b1e07b84029a6232d1bbfba3be6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:08:53.851579 17110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11957e1d5b084712b057d53625ec5e23" candidate_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "694d2b1e07b84029a6232d1bbfba3be6"
I20240501 14:08:53.851671 17110 raft_consensus.cc:3037] T 11957e1d5b084712b057d53625ec5e23 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:53.851948 17316 raft_consensus.cc:348] T f74eba74eed44278827bd37383ea632b P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.852047 17316 raft_consensus.cc:374] T f74eba74eed44278827bd37383ea632b P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.852084 17316 raft_consensus.cc:729] T f74eba74eed44278827bd37383ea632b P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 694d2b1e07b84029a6232d1bbfba3be6, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.852207 17316 consensus_queue.cc:260] T f74eba74eed44278827bd37383ea632b P 694d2b1e07b84029a6232d1bbfba3be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.852479 17316 ts_tablet_manager.cc:1398] T f74eba74eed44278827bd37383ea632b P 694d2b1e07b84029a6232d1bbfba3be6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:53.853199 17007 catalog_manager.cc:5484] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb reported cstate change: term changed from 0 to 1, leader changed from <none> to a4a6589543cb487b81dd6f0ec0a85fcb (127.16.13.194). New cstate: current_term: 1 leader_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:53.856487 17109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b3a1494abb4116ae44a9711c535435" candidate_uuid: "0333d5b1f6714097804614f828586e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "694d2b1e07b84029a6232d1bbfba3be6" is_pre_election: true
I20240501 14:08:53.856590 17109 raft_consensus.cc:2454] T e9b3a1494abb4116ae44a9711c535435 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0333d5b1f6714097804614f828586e44 in term 0.
I20240501 14:08:53.856853 17227 leader_election.cc:302] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0333d5b1f6714097804614f828586e44, 694d2b1e07b84029a6232d1bbfba3be6; no voters:
I20240501 14:08:53.857034 17311 raft_consensus.cc:2781] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:53.857091 17311 raft_consensus.cc:482] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:53.857120 17311 raft_consensus.cc:3037] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:53.857257 17186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b3a1494abb4116ae44a9711c535435" candidate_uuid: "0333d5b1f6714097804614f828586e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" is_pre_election: true
I20240501 14:08:53.857337 17186 raft_consensus.cc:2454] T e9b3a1494abb4116ae44a9711c535435 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0333d5b1f6714097804614f828586e44 in term 0.
I20240501 14:08:53.867049 17312 raft_consensus.cc:482] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:53.867134 17312 raft_consensus.cc:504] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } }
I20240501 14:08:53.867472 17312 leader_election.cc:288] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0333d5b1f6714097804614f828586e44 (127.16.13.195:35511), 694d2b1e07b84029a6232d1bbfba3be6 (127.16.13.193:33287)
I20240501 14:08:53.867717 17260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86983f2828e143df8ef2aa8f61b46353" candidate_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0333d5b1f6714097804614f828586e44" is_pre_election: true
I20240501 14:08:53.867727 17109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86983f2828e143df8ef2aa8f61b46353" candidate_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "694d2b1e07b84029a6232d1bbfba3be6" is_pre_election: true
I20240501 14:08:53.867847 17109 raft_consensus.cc:2454] T 86983f2828e143df8ef2aa8f61b46353 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4a6589543cb487b81dd6f0ec0a85fcb in term 0.
I20240501 14:08:53.867847 17260 raft_consensus.cc:2454] T 86983f2828e143df8ef2aa8f61b46353 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4a6589543cb487b81dd6f0ec0a85fcb in term 0.
I20240501 14:08:53.868116 17152 leader_election.cc:302] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 694d2b1e07b84029a6232d1bbfba3be6, a4a6589543cb487b81dd6f0ec0a85fcb; no voters:
I20240501 14:08:53.868305 17312 raft_consensus.cc:2781] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:53.868364 17312 raft_consensus.cc:482] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:53.868412 17312 raft_consensus.cc:3037] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:53.884148 17110 raft_consensus.cc:2454] T 11957e1d5b084712b057d53625ec5e23 P 694d2b1e07b84029a6232d1bbfba3be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4a6589543cb487b81dd6f0ec0a85fcb in term 1.
I20240501 14:08:53.884152 17312 raft_consensus.cc:504] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } }
I20240501 14:08:53.884547 17152 leader_election.cc:302] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 694d2b1e07b84029a6232d1bbfba3be6, a4a6589543cb487b81dd6f0ec0a85fcb; no voters:
I20240501 14:08:53.884696 17312 leader_election.cc:288] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb [CANDIDATE]: Term 1 election: Requested vote from peers 0333d5b1f6714097804614f828586e44 (127.16.13.195:35511), 694d2b1e07b84029a6232d1bbfba3be6 (127.16.13.193:33287)
I20240501 14:08:53.884800 17312 raft_consensus.cc:2781] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:53.884876 17312 raft_consensus.cc:686] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 LEADER]: Becoming Leader. State: Replica: a4a6589543cb487b81dd6f0ec0a85fcb, State: Running, Role: LEADER
I20240501 14:08:53.885035 17312 consensus_queue.cc:237] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.885213 17110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86983f2828e143df8ef2aa8f61b46353" candidate_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "694d2b1e07b84029a6232d1bbfba3be6"
I20240501 14:08:53.885325 17110 raft_consensus.cc:3037] T 86983f2828e143df8ef2aa8f61b46353 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:53.885443 17311 raft_consensus.cc:504] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.884150 17261 raft_consensus.cc:2454] T 11957e1d5b084712b057d53625ec5e23 P 0333d5b1f6714097804614f828586e44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4a6589543cb487b81dd6f0ec0a85fcb in term 1.
I20240501 14:08:53.886126 17186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b3a1494abb4116ae44a9711c535435" candidate_uuid: "0333d5b1f6714097804614f828586e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb"
I20240501 14:08:53.886222 17186 raft_consensus.cc:3037] T e9b3a1494abb4116ae44a9711c535435 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:53.886273 17110 raft_consensus.cc:2454] T 86983f2828e143df8ef2aa8f61b46353 P 694d2b1e07b84029a6232d1bbfba3be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4a6589543cb487b81dd6f0ec0a85fcb in term 1.
I20240501 14:08:53.886219 17109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b3a1494abb4116ae44a9711c535435" candidate_uuid: "0333d5b1f6714097804614f828586e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "694d2b1e07b84029a6232d1bbfba3be6"
I20240501 14:08:53.886482 17109 raft_consensus.cc:3037] T e9b3a1494abb4116ae44a9711c535435 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:53.886778 17152 leader_election.cc:302] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 694d2b1e07b84029a6232d1bbfba3be6, a4a6589543cb487b81dd6f0ec0a85fcb; no voters:
I20240501 14:08:53.886920 17332 raft_consensus.cc:2781] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:53.887089 17332 raft_consensus.cc:686] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 LEADER]: Becoming Leader. State: Replica: a4a6589543cb487b81dd6f0ec0a85fcb, State: Running, Role: LEADER
I20240501 14:08:53.887212 17332 consensus_queue.cc:237] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } }
I20240501 14:08:53.887454 17261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86983f2828e143df8ef2aa8f61b46353" candidate_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0333d5b1f6714097804614f828586e44"
I20240501 14:08:53.885772 17311 leader_election.cc:288] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44 [CANDIDATE]: Term 1 election: Requested vote from peers 694d2b1e07b84029a6232d1bbfba3be6 (127.16.13.193:33287), a4a6589543cb487b81dd6f0ec0a85fcb (127.16.13.194:32803)
I20240501 14:08:53.887552 17261 raft_consensus.cc:3037] T 86983f2828e143df8ef2aa8f61b46353 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:53.888226 17332 raft_consensus.cc:482] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:53.888306 17332 raft_consensus.cc:504] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.888502 17007 catalog_manager.cc:5484] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb reported cstate change: term changed from 0 to 1, leader changed from <none> to a4a6589543cb487b81dd6f0ec0a85fcb (127.16.13.194). New cstate: current_term: 1 leader_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:53.888622 17332 leader_election.cc:288] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 694d2b1e07b84029a6232d1bbfba3be6 (127.16.13.193:33287), 0333d5b1f6714097804614f828586e44 (127.16.13.195:35511)
I20240501 14:08:53.888860 17110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d8843520d54b7fb49be8b2cdead7b7" candidate_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "694d2b1e07b84029a6232d1bbfba3be6" is_pre_election: true
I20240501 14:08:53.888960 17110 raft_consensus.cc:2454] T 08d8843520d54b7fb49be8b2cdead7b7 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4a6589543cb487b81dd6f0ec0a85fcb in term 0.
I20240501 14:08:53.889003 17260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d8843520d54b7fb49be8b2cdead7b7" candidate_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0333d5b1f6714097804614f828586e44" is_pre_election: true
I20240501 14:08:53.889096 17260 raft_consensus.cc:2454] T 08d8843520d54b7fb49be8b2cdead7b7 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4a6589543cb487b81dd6f0ec0a85fcb in term 0.
I20240501 14:08:53.889183 17152 leader_election.cc:302] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 694d2b1e07b84029a6232d1bbfba3be6, a4a6589543cb487b81dd6f0ec0a85fcb; no voters:
I20240501 14:08:53.889317 17332 raft_consensus.cc:2781] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:53.889376 17332 raft_consensus.cc:482] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:53.889446 17332 raft_consensus.cc:3037] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:53.891350 17007 catalog_manager.cc:5484] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb reported cstate change: term changed from 0 to 1, leader changed from <none> to a4a6589543cb487b81dd6f0ec0a85fcb (127.16.13.194). New cstate: current_term: 1 leader_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:53.938454 17321 raft_consensus.cc:482] T 370d7c1dc64d40fa87489f566515bf30 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:53.938609 17321 raft_consensus.cc:504] T 370d7c1dc64d40fa87489f566515bf30 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.939159 17321 leader_election.cc:288] T 370d7c1dc64d40fa87489f566515bf30 P 694d2b1e07b84029a6232d1bbfba3be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4a6589543cb487b81dd6f0ec0a85fcb (127.16.13.194:32803), 0333d5b1f6714097804614f828586e44 (127.16.13.195:35511)
I20240501 14:08:53.942715 17185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370d7c1dc64d40fa87489f566515bf30" candidate_uuid: "694d2b1e07b84029a6232d1bbfba3be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" is_pre_election: true
I20240501 14:08:53.942867 17185 raft_consensus.cc:2454] T 370d7c1dc64d40fa87489f566515bf30 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 694d2b1e07b84029a6232d1bbfba3be6 in term 0.
I20240501 14:08:53.942890 17260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370d7c1dc64d40fa87489f566515bf30" candidate_uuid: "694d2b1e07b84029a6232d1bbfba3be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0333d5b1f6714097804614f828586e44" is_pre_election: true
I20240501 14:08:53.942970 17260 raft_consensus.cc:2454] T 370d7c1dc64d40fa87489f566515bf30 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 694d2b1e07b84029a6232d1bbfba3be6 in term 0.
I20240501 14:08:53.943187 17073 leader_election.cc:302] T 370d7c1dc64d40fa87489f566515bf30 P 694d2b1e07b84029a6232d1bbfba3be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 694d2b1e07b84029a6232d1bbfba3be6, a4a6589543cb487b81dd6f0ec0a85fcb; no voters:
I20240501 14:08:53.943351 17321 raft_consensus.cc:2781] T 370d7c1dc64d40fa87489f566515bf30 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:53.943434 17321 raft_consensus.cc:482] T 370d7c1dc64d40fa87489f566515bf30 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:53.943473 17321 raft_consensus.cc:3037] T 370d7c1dc64d40fa87489f566515bf30 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:53.974445 17186 raft_consensus.cc:2454] T e9b3a1494abb4116ae44a9711c535435 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0333d5b1f6714097804614f828586e44 in term 1.
I20240501 14:08:53.974445 17261 raft_consensus.cc:2454] T 86983f2828e143df8ef2aa8f61b46353 P 0333d5b1f6714097804614f828586e44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4a6589543cb487b81dd6f0ec0a85fcb in term 1.
I20240501 14:08:53.975358 17224 leader_election.cc:302] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0333d5b1f6714097804614f828586e44, a4a6589543cb487b81dd6f0ec0a85fcb; no voters:
I20240501 14:08:53.974445 17109 raft_consensus.cc:2454] T e9b3a1494abb4116ae44a9711c535435 P 694d2b1e07b84029a6232d1bbfba3be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0333d5b1f6714097804614f828586e44 in term 1.
I20240501 14:08:53.975682 17311 raft_consensus.cc:2781] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:53.975322 17332 raft_consensus.cc:504] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.975322 17321 raft_consensus.cc:504] T 370d7c1dc64d40fa87489f566515bf30 P 694d2b1e07b84029a6232d1bbfba3be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.976145 17332 leader_election.cc:288] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb [CANDIDATE]: Term 1 election: Requested vote from peers 694d2b1e07b84029a6232d1bbfba3be6 (127.16.13.193:33287), 0333d5b1f6714097804614f828586e44 (127.16.13.195:35511)
I20240501 14:08:53.976338 17321 leader_election.cc:288] T 370d7c1dc64d40fa87489f566515bf30 P 694d2b1e07b84029a6232d1bbfba3be6 [CANDIDATE]: Term 1 election: Requested vote from peers a4a6589543cb487b81dd6f0ec0a85fcb (127.16.13.194:32803), 0333d5b1f6714097804614f828586e44 (127.16.13.195:35511)
I20240501 14:08:53.975905 17311 raft_consensus.cc:686] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44 [term 1 LEADER]: Becoming Leader. State: Replica: 0333d5b1f6714097804614f828586e44, State: Running, Role: LEADER
I20240501 14:08:53.976594 17311 consensus_queue.cc:237] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.976626 17109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d8843520d54b7fb49be8b2cdead7b7" candidate_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "694d2b1e07b84029a6232d1bbfba3be6"
I20240501 14:08:53.976738 17109 raft_consensus.cc:3037] T 08d8843520d54b7fb49be8b2cdead7b7 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:53.976632 17186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370d7c1dc64d40fa87489f566515bf30" candidate_uuid: "694d2b1e07b84029a6232d1bbfba3be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb"
I20240501 14:08:53.977361 17186 raft_consensus.cc:3037] T 370d7c1dc64d40fa87489f566515bf30 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:53.977692 17109 raft_consensus.cc:2454] T 08d8843520d54b7fb49be8b2cdead7b7 P 694d2b1e07b84029a6232d1bbfba3be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4a6589543cb487b81dd6f0ec0a85fcb in term 1.
I20240501 14:08:53.977874 17260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d8843520d54b7fb49be8b2cdead7b7" candidate_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0333d5b1f6714097804614f828586e44"
I20240501 14:08:53.977962 17152 leader_election.cc:302] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 694d2b1e07b84029a6232d1bbfba3be6, a4a6589543cb487b81dd6f0ec0a85fcb; no voters:
I20240501 14:08:53.977969 17260 raft_consensus.cc:3037] T 08d8843520d54b7fb49be8b2cdead7b7 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:53.978540 17332 raft_consensus.cc:2781] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:53.978626 17261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370d7c1dc64d40fa87489f566515bf30" candidate_uuid: "694d2b1e07b84029a6232d1bbfba3be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0333d5b1f6714097804614f828586e44"
I20240501 14:08:53.978720 17261 raft_consensus.cc:3037] T 370d7c1dc64d40fa87489f566515bf30 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:53.978631 17332 raft_consensus.cc:686] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 LEADER]: Becoming Leader. State: Replica: a4a6589543cb487b81dd6f0ec0a85fcb, State: Running, Role: LEADER
I20240501 14:08:53.979018 17332 consensus_queue.cc:237] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:53.979028 17007 catalog_manager.cc:5484] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0333d5b1f6714097804614f828586e44 (127.16.13.195). New cstate: current_term: 1 leader_uuid: "0333d5b1f6714097804614f828586e44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } health_report { overall_health: HEALTHY } } }
I20240501 14:08:53.980406 17006 catalog_manager.cc:5484] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb reported cstate change: term changed from 0 to 1, leader changed from <none> to a4a6589543cb487b81dd6f0ec0a85fcb (127.16.13.194). New cstate: current_term: 1 leader_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:54.035576 17332 raft_consensus.cc:482] T f74eba74eed44278827bd37383ea632b P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:54.035712 17332 raft_consensus.cc:504] T f74eba74eed44278827bd37383ea632b P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:54.036162 17332 leader_election.cc:288] T f74eba74eed44278827bd37383ea632b P a4a6589543cb487b81dd6f0ec0a85fcb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 694d2b1e07b84029a6232d1bbfba3be6 (127.16.13.193:33287), 0333d5b1f6714097804614f828586e44 (127.16.13.195:35511)
I20240501 14:08:54.036470 17109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f74eba74eed44278827bd37383ea632b" candidate_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "694d2b1e07b84029a6232d1bbfba3be6" is_pre_election: true
I20240501 14:08:54.036604 17109 raft_consensus.cc:2454] T f74eba74eed44278827bd37383ea632b P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4a6589543cb487b81dd6f0ec0a85fcb in term 0.
I20240501 14:08:54.036608 17259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f74eba74eed44278827bd37383ea632b" candidate_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0333d5b1f6714097804614f828586e44" is_pre_election: true
I20240501 14:08:54.036798 17259 raft_consensus.cc:2454] T f74eba74eed44278827bd37383ea632b P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4a6589543cb487b81dd6f0ec0a85fcb in term 0.
I20240501 14:08:54.036870 17152 leader_election.cc:302] T f74eba74eed44278827bd37383ea632b P a4a6589543cb487b81dd6f0ec0a85fcb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 694d2b1e07b84029a6232d1bbfba3be6, a4a6589543cb487b81dd6f0ec0a85fcb; no voters:
I20240501 14:08:54.037047 17332 raft_consensus.cc:2781] T f74eba74eed44278827bd37383ea632b P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:54.037101 17332 raft_consensus.cc:482] T f74eba74eed44278827bd37383ea632b P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:54.037129 17332 raft_consensus.cc:3037] T f74eba74eed44278827bd37383ea632b P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:54.067652 17186 raft_consensus.cc:2454] T 370d7c1dc64d40fa87489f566515bf30 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 694d2b1e07b84029a6232d1bbfba3be6 in term 1.
I20240501 14:08:54.067747 17261 raft_consensus.cc:2454] T 370d7c1dc64d40fa87489f566515bf30 P 0333d5b1f6714097804614f828586e44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 694d2b1e07b84029a6232d1bbfba3be6 in term 1.
I20240501 14:08:54.067652 17260 raft_consensus.cc:2454] T 08d8843520d54b7fb49be8b2cdead7b7 P 0333d5b1f6714097804614f828586e44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4a6589543cb487b81dd6f0ec0a85fcb in term 1.
I20240501 14:08:54.068169 17075 leader_election.cc:302] T 370d7c1dc64d40fa87489f566515bf30 P 694d2b1e07b84029a6232d1bbfba3be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0333d5b1f6714097804614f828586e44, 694d2b1e07b84029a6232d1bbfba3be6; no voters:
I20240501 14:08:54.068409 17321 raft_consensus.cc:2781] T 370d7c1dc64d40fa87489f566515bf30 P 694d2b1e07b84029a6232d1bbfba3be6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:54.068653 17321 raft_consensus.cc:686] T 370d7c1dc64d40fa87489f566515bf30 P 694d2b1e07b84029a6232d1bbfba3be6 [term 1 LEADER]: Becoming Leader. State: Replica: 694d2b1e07b84029a6232d1bbfba3be6, State: Running, Role: LEADER
I20240501 14:08:54.068841 17321 consensus_queue.cc:237] T 370d7c1dc64d40fa87489f566515bf30 P 694d2b1e07b84029a6232d1bbfba3be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:54.069435 17332 raft_consensus.cc:504] T f74eba74eed44278827bd37383ea632b P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:54.070053 17109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f74eba74eed44278827bd37383ea632b" candidate_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "694d2b1e07b84029a6232d1bbfba3be6"
I20240501 14:08:54.070163 17109 raft_consensus.cc:3037] T f74eba74eed44278827bd37383ea632b P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:54.070478 17332 leader_election.cc:288] T f74eba74eed44278827bd37383ea632b P a4a6589543cb487b81dd6f0ec0a85fcb [CANDIDATE]: Term 1 election: Requested vote from peers 694d2b1e07b84029a6232d1bbfba3be6 (127.16.13.193:33287), 0333d5b1f6714097804614f828586e44 (127.16.13.195:35511)
I20240501 14:08:54.070591 17006 catalog_manager.cc:5484] T 370d7c1dc64d40fa87489f566515bf30 P 694d2b1e07b84029a6232d1bbfba3be6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 694d2b1e07b84029a6232d1bbfba3be6 (127.16.13.193). New cstate: current_term: 1 leader_uuid: "694d2b1e07b84029a6232d1bbfba3be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:54.070681 17260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f74eba74eed44278827bd37383ea632b" candidate_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0333d5b1f6714097804614f828586e44"
I20240501 14:08:54.070771 17260 raft_consensus.cc:3037] T f74eba74eed44278827bd37383ea632b P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:54.113943 17332 raft_consensus.cc:482] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:54.114095 17332 raft_consensus.cc:504] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:54.114493 17332 leader_election.cc:288] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 694d2b1e07b84029a6232d1bbfba3be6 (127.16.13.193:33287), 0333d5b1f6714097804614f828586e44 (127.16.13.195:35511)
I20240501 14:08:54.115022 17110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6e8031453c845e083866e9706373eb8" candidate_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "694d2b1e07b84029a6232d1bbfba3be6" is_pre_election: true
I20240501 14:08:54.115022 17261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6e8031453c845e083866e9706373eb8" candidate_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0333d5b1f6714097804614f828586e44" is_pre_election: true
I20240501 14:08:54.115198 17261 raft_consensus.cc:2454] T b6e8031453c845e083866e9706373eb8 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4a6589543cb487b81dd6f0ec0a85fcb in term 0.
I20240501 14:08:54.115198 17110 raft_consensus.cc:2454] T b6e8031453c845e083866e9706373eb8 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4a6589543cb487b81dd6f0ec0a85fcb in term 0.
I20240501 14:08:54.115571 17151 leader_election.cc:302] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0333d5b1f6714097804614f828586e44, a4a6589543cb487b81dd6f0ec0a85fcb; no voters:
I20240501 14:08:54.115772 17332 raft_consensus.cc:2781] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:54.115846 17332 raft_consensus.cc:482] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:54.115900 17332 raft_consensus.cc:3037] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:54.117985 17109 raft_consensus.cc:2454] T f74eba74eed44278827bd37383ea632b P 694d2b1e07b84029a6232d1bbfba3be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4a6589543cb487b81dd6f0ec0a85fcb in term 1.
I20240501 14:08:54.118014 17260 raft_consensus.cc:2454] T f74eba74eed44278827bd37383ea632b P 0333d5b1f6714097804614f828586e44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4a6589543cb487b81dd6f0ec0a85fcb in term 1.
I20240501 14:08:54.117990 17332 raft_consensus.cc:504] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:54.118432 17332 leader_election.cc:288] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb [CANDIDATE]: Term 1 election: Requested vote from peers 694d2b1e07b84029a6232d1bbfba3be6 (127.16.13.193:33287), 0333d5b1f6714097804614f828586e44 (127.16.13.195:35511)
I20240501 14:08:54.118432 17152 leader_election.cc:302] T f74eba74eed44278827bd37383ea632b P a4a6589543cb487b81dd6f0ec0a85fcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 694d2b1e07b84029a6232d1bbfba3be6, a4a6589543cb487b81dd6f0ec0a85fcb; no voters:
I20240501 14:08:54.118674 17332 raft_consensus.cc:2781] T f74eba74eed44278827bd37383ea632b P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:54.118814 17332 raft_consensus.cc:686] T f74eba74eed44278827bd37383ea632b P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 LEADER]: Becoming Leader. State: Replica: a4a6589543cb487b81dd6f0ec0a85fcb, State: Running, Role: LEADER
I20240501 14:08:54.118935 17109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6e8031453c845e083866e9706373eb8" candidate_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "694d2b1e07b84029a6232d1bbfba3be6"
I20240501 14:08:54.118959 17260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6e8031453c845e083866e9706373eb8" candidate_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0333d5b1f6714097804614f828586e44"
I20240501 14:08:54.119058 17109 raft_consensus.cc:3037] T b6e8031453c845e083866e9706373eb8 P 694d2b1e07b84029a6232d1bbfba3be6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:54.119062 17260 raft_consensus.cc:3037] T b6e8031453c845e083866e9706373eb8 P 0333d5b1f6714097804614f828586e44 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:54.119086 17332 consensus_queue.cc:237] T f74eba74eed44278827bd37383ea632b P a4a6589543cb487b81dd6f0ec0a85fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:54.119953 17260 raft_consensus.cc:2454] T b6e8031453c845e083866e9706373eb8 P 0333d5b1f6714097804614f828586e44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4a6589543cb487b81dd6f0ec0a85fcb in term 1.
I20240501 14:08:54.119953 17109 raft_consensus.cc:2454] T b6e8031453c845e083866e9706373eb8 P 694d2b1e07b84029a6232d1bbfba3be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4a6589543cb487b81dd6f0ec0a85fcb in term 1.
I20240501 14:08:54.120241 17152 leader_election.cc:302] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 694d2b1e07b84029a6232d1bbfba3be6, a4a6589543cb487b81dd6f0ec0a85fcb; no voters:
I20240501 14:08:54.120460 17344 raft_consensus.cc:2781] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:54.120546 17344 raft_consensus.cc:686] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 LEADER]: Becoming Leader. State: Replica: a4a6589543cb487b81dd6f0ec0a85fcb, State: Running, Role: LEADER
I20240501 14:08:54.120690 17344 consensus_queue.cc:237] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:54.122392 17006 catalog_manager.cc:5484] T f74eba74eed44278827bd37383ea632b P a4a6589543cb487b81dd6f0ec0a85fcb reported cstate change: term changed from 0 to 1, leader changed from <none> to a4a6589543cb487b81dd6f0ec0a85fcb (127.16.13.194). New cstate: current_term: 1 leader_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:54.125746 17006 catalog_manager.cc:5484] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb reported cstate change: term changed from 0 to 1, leader changed from <none> to a4a6589543cb487b81dd6f0ec0a85fcb (127.16.13.194). New cstate: current_term: 1 leader_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:54.163463 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:54.166200 17358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:54.166989 17355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:54.167637 17356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:54.168406 16439 server_base.cc:1034] running on GCE node
I20240501 14:08:54.168584 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:08:54.168623 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:08:54.168646 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572534168646 us; error 0 us; skew 500 ppm
I20240501 14:08:54.169636 16439 webserver.cc:460] Webserver started at http://127.16.13.196:38763/ using document root <none> and password file <none>
I20240501 14:08:54.169785 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:54.169838 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:54.169917 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:54.170336 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714572529860089-16439-0/minicluster-data/ts-3-root/instance:
uuid: "547ad28f9bbf4ee2835750ec16680214"
format_stamp: "Formatted at 2024-05-01 14:08:54 on dist-test-slave-mgxc"
I20240501 14:08:54.171734 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:54.172688 17363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:54.172902 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:54.172969 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714572529860089-16439-0/minicluster-data/ts-3-root
uuid: "547ad28f9bbf4ee2835750ec16680214"
format_stamp: "Formatted at 2024-05-01 14:08:54 on dist-test-slave-mgxc"
I20240501 14:08:54.173034 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714572529860089-16439-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714572529860089-16439-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714572529860089-16439-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:54.181008 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:54.181358 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:54.181638 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:54.182054 16439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:54.182096 16439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:54.182152 16439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:54.182179 16439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:54.187057 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.196:32965
I20240501 14:08:54.187115 17426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.196:32965 every 8 connection(s)
I20240501 14:08:54.191509 17427 heartbeater.cc:347] Connected to a master server at 127.16.13.254:42089
I20240501 14:08:54.191593 17427 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:54.191793 17427 heartbeater.cc:511] Master 127.16.13.254:42089 requested a full tablet report, sending...
I20240501 14:08:54.192425 17006 ts_manager.cc:194] Registered new tserver with Master: 547ad28f9bbf4ee2835750ec16680214 (127.16.13.196:32965)
I20240501 14:08:54.192921 17006 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60044
I20240501 14:08:54.284538 17344 consensus_queue.cc:1035] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:54.287981 17312 consensus_queue.cc:1035] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:54.331431 17312 consensus_queue.cc:1035] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:54.334484 17332 consensus_queue.cc:1035] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:54.418409 17332 consensus_queue.cc:1035] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:54.418419 17311 consensus_queue.cc:1035] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:54.421557 17348 consensus_queue.cc:1035] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:54.423427 17312 consensus_queue.cc:1035] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:54.463992 17312 consensus_queue.cc:1035] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:54.467079 17312 consensus_queue.cc:1035] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:54.530416 17312 consensus_queue.cc:1035] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:54.533478 17332 consensus_queue.cc:1035] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:54.599869 17351 consensus_queue.cc:1035] T 370d7c1dc64d40fa87489f566515bf30 P 694d2b1e07b84029a6232d1bbfba3be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:54.602989 17351 consensus_queue.cc:1035] T 370d7c1dc64d40fa87489f566515bf30 P 694d2b1e07b84029a6232d1bbfba3be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:54.615263 17332 consensus_queue.cc:1035] T f74eba74eed44278827bd37383ea632b P a4a6589543cb487b81dd6f0ec0a85fcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:54.618047 17442 consensus_queue.cc:1035] T f74eba74eed44278827bd37383ea632b P a4a6589543cb487b81dd6f0ec0a85fcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:54.621668 17061 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:08:54.627511 17186 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "08d8843520d54b7fb49be8b2cdead7b7"
dest_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb"
mode: GRACEFUL
new_leader_uuid: "694d2b1e07b84029a6232d1bbfba3be6"
from {username='slave'} at 127.0.0.1:54488
I20240501 14:08:54.627666 17186 raft_consensus.cc:595] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 LEADER]: Received request to transfer leadership to TS 694d2b1e07b84029a6232d1bbfba3be6
I20240501 14:08:54.628494 17186 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "86983f2828e143df8ef2aa8f61b46353"
dest_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb"
mode: GRACEFUL
new_leader_uuid: "0333d5b1f6714097804614f828586e44"
from {username='slave'} at 127.0.0.1:54488
I20240501 14:08:54.628593 17186 raft_consensus.cc:595] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 LEADER]: Received request to transfer leadership to TS 0333d5b1f6714097804614f828586e44
I20240501 14:08:54.629259 17186 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b6e8031453c845e083866e9706373eb8"
dest_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb"
mode: GRACEFUL
new_leader_uuid: "694d2b1e07b84029a6232d1bbfba3be6"
from {username='slave'} at 127.0.0.1:54488
I20240501 14:08:54.629346 17186 raft_consensus.cc:595] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 LEADER]: Received request to transfer leadership to TS 694d2b1e07b84029a6232d1bbfba3be6
I20240501 14:08:54.629631 17061 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240501 14:08:54.629823 17061 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:08:54.753340 17312 raft_consensus.cc:979] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb: : Instructing follower 0333d5b1f6714097804614f828586e44 to start an election
I20240501 14:08:54.753484 17312 raft_consensus.cc:1067] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 LEADER]: Signalling peer 0333d5b1f6714097804614f828586e44 to start an election
I20240501 14:08:54.754069 17260 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "86983f2828e143df8ef2aa8f61b46353"
dest_uuid: "0333d5b1f6714097804614f828586e44"
from {username='slave'} at 127.0.0.1:35216
I20240501 14:08:54.754213 17260 raft_consensus.cc:482] T 86983f2828e143df8ef2aa8f61b46353 P 0333d5b1f6714097804614f828586e44 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:08:54.754256 17260 raft_consensus.cc:3037] T 86983f2828e143df8ef2aa8f61b46353 P 0333d5b1f6714097804614f828586e44 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:08:54.827395 17260 raft_consensus.cc:504] T 86983f2828e143df8ef2aa8f61b46353 P 0333d5b1f6714097804614f828586e44 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } }
I20240501 14:08:54.827818 17260 leader_election.cc:288] T 86983f2828e143df8ef2aa8f61b46353 P 0333d5b1f6714097804614f828586e44 [CANDIDATE]: Term 2 election: Requested vote from peers a4a6589543cb487b81dd6f0ec0a85fcb (127.16.13.194:32803), 694d2b1e07b84029a6232d1bbfba3be6 (127.16.13.193:33287)
I20240501 14:08:54.828183 17186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86983f2828e143df8ef2aa8f61b46353" candidate_uuid: "0333d5b1f6714097804614f828586e44" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb"
I20240501 14:08:54.828204 17109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86983f2828e143df8ef2aa8f61b46353" candidate_uuid: "0333d5b1f6714097804614f828586e44" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "694d2b1e07b84029a6232d1bbfba3be6"
I20240501 14:08:54.828333 17186 raft_consensus.cc:3032] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:08:54.828349 17109 raft_consensus.cc:3037] T 86983f2828e143df8ef2aa8f61b46353 P 694d2b1e07b84029a6232d1bbfba3be6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:08:54.828397 17186 raft_consensus.cc:729] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a4a6589543cb487b81dd6f0ec0a85fcb, State: Running, Role: LEADER
I20240501 14:08:54.828573 17186 consensus_queue.cc:260] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } }
I20240501 14:08:54.828779 17186 raft_consensus.cc:3037] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:08:54.829257 17109 raft_consensus.cc:2454] T 86983f2828e143df8ef2aa8f61b46353 P 694d2b1e07b84029a6232d1bbfba3be6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0333d5b1f6714097804614f828586e44 in term 2.
I20240501 14:08:54.829522 17227 leader_election.cc:302] T 86983f2828e143df8ef2aa8f61b46353 P 0333d5b1f6714097804614f828586e44 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0333d5b1f6714097804614f828586e44, 694d2b1e07b84029a6232d1bbfba3be6; no voters:
I20240501 14:08:54.829685 17348 raft_consensus.cc:2781] T 86983f2828e143df8ef2aa8f61b46353 P 0333d5b1f6714097804614f828586e44 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:08:54.829758 17348 raft_consensus.cc:686] T 86983f2828e143df8ef2aa8f61b46353 P 0333d5b1f6714097804614f828586e44 [term 2 LEADER]: Becoming Leader. State: Replica: 0333d5b1f6714097804614f828586e44, State: Running, Role: LEADER
I20240501 14:08:54.829883 17348 consensus_queue.cc:237] T 86983f2828e143df8ef2aa8f61b46353 P 0333d5b1f6714097804614f828586e44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } }
I20240501 14:08:54.829903 17186 raft_consensus.cc:2454] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0333d5b1f6714097804614f828586e44 in term 2.
I20240501 14:08:54.831305 17007 catalog_manager.cc:5484] T 86983f2828e143df8ef2aa8f61b46353 P 0333d5b1f6714097804614f828586e44 reported cstate change: term changed from 1 to 2, leader changed from a4a6589543cb487b81dd6f0ec0a85fcb (127.16.13.194) to 0333d5b1f6714097804614f828586e44 (127.16.13.195). New cstate: current_term: 2 leader_uuid: "0333d5b1f6714097804614f828586e44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:54.998651 17452 raft_consensus.cc:979] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb: : Instructing follower 694d2b1e07b84029a6232d1bbfba3be6 to start an election
I20240501 14:08:54.998816 17452 raft_consensus.cc:1067] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 LEADER]: Signalling peer 694d2b1e07b84029a6232d1bbfba3be6 to start an election
I20240501 14:08:54.999300 17109 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b6e8031453c845e083866e9706373eb8"
dest_uuid: "694d2b1e07b84029a6232d1bbfba3be6"
from {username='slave'} at 127.0.0.1:60692
I20240501 14:08:54.999480 17109 raft_consensus.cc:482] T b6e8031453c845e083866e9706373eb8 P 694d2b1e07b84029a6232d1bbfba3be6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:08:54.999550 17109 raft_consensus.cc:3037] T b6e8031453c845e083866e9706373eb8 P 694d2b1e07b84029a6232d1bbfba3be6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:08:55.017884 17312 raft_consensus.cc:979] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb: : Instructing follower 694d2b1e07b84029a6232d1bbfba3be6 to start an election
I20240501 14:08:55.018000 17312 raft_consensus.cc:1067] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 LEADER]: Signalling peer 694d2b1e07b84029a6232d1bbfba3be6 to start an election
I20240501 14:08:55.018374 17110 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "08d8843520d54b7fb49be8b2cdead7b7"
dest_uuid: "694d2b1e07b84029a6232d1bbfba3be6"
from {username='slave'} at 127.0.0.1:60692
I20240501 14:08:55.018536 17110 raft_consensus.cc:482] T 08d8843520d54b7fb49be8b2cdead7b7 P 694d2b1e07b84029a6232d1bbfba3be6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:08:55.018610 17110 raft_consensus.cc:3037] T 08d8843520d54b7fb49be8b2cdead7b7 P 694d2b1e07b84029a6232d1bbfba3be6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:08:55.023116 17109 raft_consensus.cc:504] T b6e8031453c845e083866e9706373eb8 P 694d2b1e07b84029a6232d1bbfba3be6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:55.023509 17109 leader_election.cc:288] T b6e8031453c845e083866e9706373eb8 P 694d2b1e07b84029a6232d1bbfba3be6 [CANDIDATE]: Term 2 election: Requested vote from peers a4a6589543cb487b81dd6f0ec0a85fcb (127.16.13.194:32803), 0333d5b1f6714097804614f828586e44 (127.16.13.195:35511)
I20240501 14:08:55.023846 17186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6e8031453c845e083866e9706373eb8" candidate_uuid: "694d2b1e07b84029a6232d1bbfba3be6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb"
I20240501 14:08:55.023974 17186 raft_consensus.cc:3032] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:08:55.024016 17186 raft_consensus.cc:729] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a4a6589543cb487b81dd6f0ec0a85fcb, State: Running, Role: LEADER
I20240501 14:08:55.024147 17260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6e8031453c845e083866e9706373eb8" candidate_uuid: "694d2b1e07b84029a6232d1bbfba3be6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0333d5b1f6714097804614f828586e44"
I20240501 14:08:55.024174 17186 consensus_queue.cc:260] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:55.024241 17260 raft_consensus.cc:3037] T b6e8031453c845e083866e9706373eb8 P 0333d5b1f6714097804614f828586e44 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:08:55.024369 17186 raft_consensus.cc:3037] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:08:55.024222 17110 raft_consensus.cc:504] T 08d8843520d54b7fb49be8b2cdead7b7 P 694d2b1e07b84029a6232d1bbfba3be6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:55.024955 17110 leader_election.cc:288] T 08d8843520d54b7fb49be8b2cdead7b7 P 694d2b1e07b84029a6232d1bbfba3be6 [CANDIDATE]: Term 2 election: Requested vote from peers a4a6589543cb487b81dd6f0ec0a85fcb (127.16.13.194:32803), 0333d5b1f6714097804614f828586e44 (127.16.13.195:35511)
I20240501 14:08:55.025166 17186 raft_consensus.cc:2454] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 694d2b1e07b84029a6232d1bbfba3be6 in term 2.
I20240501 14:08:55.025259 17261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d8843520d54b7fb49be8b2cdead7b7" candidate_uuid: "694d2b1e07b84029a6232d1bbfba3be6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0333d5b1f6714097804614f828586e44"
I20240501 14:08:55.025342 17261 raft_consensus.cc:3037] T 08d8843520d54b7fb49be8b2cdead7b7 P 0333d5b1f6714097804614f828586e44 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:08:55.025605 17073 leader_election.cc:302] T b6e8031453c845e083866e9706373eb8 P 694d2b1e07b84029a6232d1bbfba3be6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 694d2b1e07b84029a6232d1bbfba3be6, a4a6589543cb487b81dd6f0ec0a85fcb; no voters:
I20240501 14:08:55.025780 17186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d8843520d54b7fb49be8b2cdead7b7" candidate_uuid: "694d2b1e07b84029a6232d1bbfba3be6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb"
I20240501 14:08:55.025866 17186 raft_consensus.cc:3032] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:08:55.025916 17186 raft_consensus.cc:729] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a4a6589543cb487b81dd6f0ec0a85fcb, State: Running, Role: LEADER
I20240501 14:08:55.025919 17351 raft_consensus.cc:2781] T b6e8031453c845e083866e9706373eb8 P 694d2b1e07b84029a6232d1bbfba3be6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:08:55.026047 17351 raft_consensus.cc:686] T b6e8031453c845e083866e9706373eb8 P 694d2b1e07b84029a6232d1bbfba3be6 [term 2 LEADER]: Becoming Leader. State: Replica: 694d2b1e07b84029a6232d1bbfba3be6, State: Running, Role: LEADER
I20240501 14:08:55.026211 17351 consensus_queue.cc:237] T b6e8031453c845e083866e9706373eb8 P 694d2b1e07b84029a6232d1bbfba3be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:55.026053 17186 consensus_queue.cc:260] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:55.026448 17186 raft_consensus.cc:3037] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:08:55.027673 17007 catalog_manager.cc:5484] T b6e8031453c845e083866e9706373eb8 P 694d2b1e07b84029a6232d1bbfba3be6 reported cstate change: term changed from 1 to 2, leader changed from a4a6589543cb487b81dd6f0ec0a85fcb (127.16.13.194) to 694d2b1e07b84029a6232d1bbfba3be6 (127.16.13.193). New cstate: current_term: 2 leader_uuid: "694d2b1e07b84029a6232d1bbfba3be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:55.120182 17186 raft_consensus.cc:2454] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 694d2b1e07b84029a6232d1bbfba3be6 in term 2.
I20240501 14:08:55.120182 17260 raft_consensus.cc:2454] T b6e8031453c845e083866e9706373eb8 P 0333d5b1f6714097804614f828586e44 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 694d2b1e07b84029a6232d1bbfba3be6 in term 2.
I20240501 14:08:55.120285 17261 raft_consensus.cc:2454] T 08d8843520d54b7fb49be8b2cdead7b7 P 0333d5b1f6714097804614f828586e44 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 694d2b1e07b84029a6232d1bbfba3be6 in term 2.
I20240501 14:08:55.120783 17073 leader_election.cc:302] T 08d8843520d54b7fb49be8b2cdead7b7 P 694d2b1e07b84029a6232d1bbfba3be6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 694d2b1e07b84029a6232d1bbfba3be6, a4a6589543cb487b81dd6f0ec0a85fcb; no voters:
I20240501 14:08:55.121023 17351 raft_consensus.cc:2781] T 08d8843520d54b7fb49be8b2cdead7b7 P 694d2b1e07b84029a6232d1bbfba3be6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:08:55.121111 17351 raft_consensus.cc:686] T 08d8843520d54b7fb49be8b2cdead7b7 P 694d2b1e07b84029a6232d1bbfba3be6 [term 2 LEADER]: Becoming Leader. State: Replica: 694d2b1e07b84029a6232d1bbfba3be6, State: Running, Role: LEADER
I20240501 14:08:55.121311 17351 consensus_queue.cc:237] T 08d8843520d54b7fb49be8b2cdead7b7 P 694d2b1e07b84029a6232d1bbfba3be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } }
I20240501 14:08:55.123172 17007 catalog_manager.cc:5484] T 08d8843520d54b7fb49be8b2cdead7b7 P 694d2b1e07b84029a6232d1bbfba3be6 reported cstate change: term changed from 1 to 2, leader changed from a4a6589543cb487b81dd6f0ec0a85fcb (127.16.13.194) to 694d2b1e07b84029a6232d1bbfba3be6 (127.16.13.193). New cstate: current_term: 2 leader_uuid: "694d2b1e07b84029a6232d1bbfba3be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:55.193730 17427 heartbeater.cc:503] Master 127.16.13.254:42089 was elected leader, sending a full tablet report...
I20240501 14:08:55.328408 17110 raft_consensus.cc:1261] T 86983f2828e143df8ef2aa8f61b46353 P 694d2b1e07b84029a6232d1bbfba3be6 [term 2 FOLLOWER]: Refusing update from remote peer 0333d5b1f6714097804614f828586e44: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:08:55.328938 17456 consensus_queue.cc:1035] T 86983f2828e143df8ef2aa8f61b46353 P 0333d5b1f6714097804614f828586e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:08:55.331251 17186 raft_consensus.cc:1261] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb [term 2 FOLLOWER]: Refusing update from remote peer 0333d5b1f6714097804614f828586e44: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:08:55.331804 17456 consensus_queue.cc:1035] T 86983f2828e143df8ef2aa8f61b46353 P 0333d5b1f6714097804614f828586e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:08:55.405958 17186 raft_consensus.cc:1261] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb [term 2 FOLLOWER]: Refusing update from remote peer 694d2b1e07b84029a6232d1bbfba3be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:08:55.406430 17457 consensus_queue.cc:1035] T b6e8031453c845e083866e9706373eb8 P 694d2b1e07b84029a6232d1bbfba3be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:08:55.409044 17260 raft_consensus.cc:1261] T b6e8031453c845e083866e9706373eb8 P 0333d5b1f6714097804614f828586e44 [term 2 FOLLOWER]: Refusing update from remote peer 694d2b1e07b84029a6232d1bbfba3be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:08:55.409417 17457 consensus_queue.cc:1035] T b6e8031453c845e083866e9706373eb8 P 694d2b1e07b84029a6232d1bbfba3be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:08:55.542603 17260 raft_consensus.cc:1261] T 08d8843520d54b7fb49be8b2cdead7b7 P 0333d5b1f6714097804614f828586e44 [term 2 FOLLOWER]: Refusing update from remote peer 694d2b1e07b84029a6232d1bbfba3be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:08:55.543006 17457 consensus_queue.cc:1035] T 08d8843520d54b7fb49be8b2cdead7b7 P 694d2b1e07b84029a6232d1bbfba3be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:08:55.545522 17186 raft_consensus.cc:1261] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb [term 2 FOLLOWER]: Refusing update from remote peer 694d2b1e07b84029a6232d1bbfba3be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:08:55.545866 17462 consensus_queue.cc:1035] T 08d8843520d54b7fb49be8b2cdead7b7 P 694d2b1e07b84029a6232d1bbfba3be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:08:56.630185 17061 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:08:56.630987 17061 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:08:56.631058 17061 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:08:58.631330 17061 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:08:58.632129 17061 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:08:58.632203 17061 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:08:59.636734 17186 consensus_queue.cc:237] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } attrs { replace: true } } peers { permanent_uuid: "547ad28f9bbf4ee2835750ec16680214" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 32965 } attrs { promote: true } }
I20240501 14:08:59.638384 17260 raft_consensus.cc:1261] T eb2837529c914f5192cdaf3fae16afc1 P 0333d5b1f6714097804614f828586e44 [term 1 FOLLOWER]: Refusing update from remote peer a4a6589543cb487b81dd6f0ec0a85fcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:59.638445 17110 raft_consensus.cc:1261] T eb2837529c914f5192cdaf3fae16afc1 P 694d2b1e07b84029a6232d1bbfba3be6 [term 1 FOLLOWER]: Refusing update from remote peer a4a6589543cb487b81dd6f0ec0a85fcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:59.638772 17468 consensus_queue.cc:1035] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:08:59.639039 17467 consensus_queue.cc:1035] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:08:59.640756 17442 raft_consensus.cc:2932] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 547ad28f9bbf4ee2835750ec16680214 (127.16.13.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } attrs { replace: true } } peers { permanent_uuid: "547ad28f9bbf4ee2835750ec16680214" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 32965 } attrs { promote: true } } }
I20240501 14:08:59.641147 17260 raft_consensus.cc:2932] T eb2837529c914f5192cdaf3fae16afc1 P 0333d5b1f6714097804614f828586e44 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 547ad28f9bbf4ee2835750ec16680214 (127.16.13.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } attrs { replace: true } } peers { permanent_uuid: "547ad28f9bbf4ee2835750ec16680214" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 32965 } attrs { promote: true } } }
I20240501 14:08:59.641170 17110 raft_consensus.cc:2932] T eb2837529c914f5192cdaf3fae16afc1 P 694d2b1e07b84029a6232d1bbfba3be6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 547ad28f9bbf4ee2835750ec16680214 (127.16.13.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } attrs { replace: true } } peers { permanent_uuid: "547ad28f9bbf4ee2835750ec16680214" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 32965 } attrs { promote: true } } }
I20240501 14:08:59.643390 17007 catalog_manager.cc:5484] T eb2837529c914f5192cdaf3fae16afc1 P 0333d5b1f6714097804614f828586e44 reported cstate change: config changed from index -1 to 2, NON_VOTER 547ad28f9bbf4ee2835750ec16680214 (127.16.13.196) added. New cstate: current_term: 1 leader_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } attrs { replace: true } } peers { permanent_uuid: "547ad28f9bbf4ee2835750ec16680214" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 32965 } attrs { promote: true } } }
W20240501 14:08:59.643558 17152 consensus_peers.cc:485] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb -> Peer 547ad28f9bbf4ee2835750ec16680214 (127.16.13.196:32965): Couldn't send request to peer 547ad28f9bbf4ee2835750ec16680214. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb2837529c914f5192cdaf3fae16afc1. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:08:59.648222 17260 consensus_queue.cc:237] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } attrs { replace: true } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } } peers { permanent_uuid: "547ad28f9bbf4ee2835750ec16680214" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 32965 } attrs { promote: true } }
I20240501 14:08:59.649631 17186 raft_consensus.cc:1261] T e9b3a1494abb4116ae44a9711c535435 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 FOLLOWER]: Refusing update from remote peer 0333d5b1f6714097804614f828586e44: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:59.649789 17110 raft_consensus.cc:1261] T e9b3a1494abb4116ae44a9711c535435 P 694d2b1e07b84029a6232d1bbfba3be6 [term 1 FOLLOWER]: Refusing update from remote peer 0333d5b1f6714097804614f828586e44: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:59.650086 17480 consensus_queue.cc:1035] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:08:59.650290 17456 consensus_queue.cc:1035] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:08:59.651748 17473 raft_consensus.cc:2932] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 547ad28f9bbf4ee2835750ec16680214 (127.16.13.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } attrs { replace: true } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } } peers { permanent_uuid: "547ad28f9bbf4ee2835750ec16680214" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 32965 } attrs { promote: true } } }
I20240501 14:08:59.652240 17186 raft_consensus.cc:2932] T e9b3a1494abb4116ae44a9711c535435 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 547ad28f9bbf4ee2835750ec16680214 (127.16.13.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } attrs { replace: true } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } } peers { permanent_uuid: "547ad28f9bbf4ee2835750ec16680214" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 32965 } attrs { promote: true } } }
I20240501 14:08:59.653425 17110 raft_consensus.cc:2932] T e9b3a1494abb4116ae44a9711c535435 P 694d2b1e07b84029a6232d1bbfba3be6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 547ad28f9bbf4ee2835750ec16680214 (127.16.13.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } attrs { replace: true } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } } peers { permanent_uuid: "547ad28f9bbf4ee2835750ec16680214" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 32965 } attrs { promote: true } } }
W20240501 14:08:59.654192 17227 consensus_peers.cc:485] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44 -> Peer 547ad28f9bbf4ee2835750ec16680214 (127.16.13.196:32965): Couldn't send request to peer 547ad28f9bbf4ee2835750ec16680214. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e9b3a1494abb4116ae44a9711c535435. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:08:59.654832 17186 consensus_queue.cc:237] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } attrs { replace: true } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } } peers { permanent_uuid: "547ad28f9bbf4ee2835750ec16680214" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 32965 } attrs { promote: true } }
I20240501 14:08:59.654795 17006 catalog_manager.cc:5484] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44 reported cstate change: config changed from index -1 to 2, NON_VOTER 547ad28f9bbf4ee2835750ec16680214 (127.16.13.196) added. New cstate: current_term: 1 leader_uuid: "0333d5b1f6714097804614f828586e44" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "547ad28f9bbf4ee2835750ec16680214" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 32965 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:08:59.656347 17152 consensus_peers.cc:485] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb -> Peer 547ad28f9bbf4ee2835750ec16680214 (127.16.13.196:32965): Couldn't send request to peer 547ad28f9bbf4ee2835750ec16680214. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11957e1d5b084712b057d53625ec5e23. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:08:59.656386 17110 raft_consensus.cc:1261] T 11957e1d5b084712b057d53625ec5e23 P 694d2b1e07b84029a6232d1bbfba3be6 [term 1 FOLLOWER]: Refusing update from remote peer a4a6589543cb487b81dd6f0ec0a85fcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:59.656919 17467 consensus_queue.cc:1035] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:08:59.658035 17260 raft_consensus.cc:1261] T 11957e1d5b084712b057d53625ec5e23 P 0333d5b1f6714097804614f828586e44 [term 1 FOLLOWER]: Refusing update from remote peer a4a6589543cb487b81dd6f0ec0a85fcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:59.658413 17467 consensus_queue.cc:1035] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:08:59.658694 17468 raft_consensus.cc:2932] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 547ad28f9bbf4ee2835750ec16680214 (127.16.13.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } attrs { replace: true } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } } peers { permanent_uuid: "547ad28f9bbf4ee2835750ec16680214" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 32965 } attrs { promote: true } } }
I20240501 14:08:59.659093 17110 raft_consensus.cc:2932] T 11957e1d5b084712b057d53625ec5e23 P 694d2b1e07b84029a6232d1bbfba3be6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 547ad28f9bbf4ee2835750ec16680214 (127.16.13.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } attrs { replace: true } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } } peers { permanent_uuid: "547ad28f9bbf4ee2835750ec16680214" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 32965 } attrs { promote: true } } }
I20240501 14:08:59.660801 17260 raft_consensus.cc:2932] T 11957e1d5b084712b057d53625ec5e23 P 0333d5b1f6714097804614f828586e44 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 547ad28f9bbf4ee2835750ec16680214 (127.16.13.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } attrs { replace: true } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } } peers { permanent_uuid: "547ad28f9bbf4ee2835750ec16680214" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 32965 } attrs { promote: true } } }
I20240501 14:08:59.660949 17007 catalog_manager.cc:5484] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb reported cstate change: config changed from index -1 to 2, NON_VOTER 547ad28f9bbf4ee2835750ec16680214 (127.16.13.196) added. New cstate: current_term: 1 leader_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "547ad28f9bbf4ee2835750ec16680214" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 32965 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:59.666487 17110 consensus_queue.cc:237] T b6e8031453c845e083866e9706373eb8 P 694d2b1e07b84029a6232d1bbfba3be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } attrs { replace: true } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } } peers { permanent_uuid: "547ad28f9bbf4ee2835750ec16680214" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 32965 } attrs { promote: true } }
I20240501 14:08:59.667762 17260 raft_consensus.cc:1261] T b6e8031453c845e083866e9706373eb8 P 0333d5b1f6714097804614f828586e44 [term 2 FOLLOWER]: Refusing update from remote peer 694d2b1e07b84029a6232d1bbfba3be6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 14:08:59.667868 17186 raft_consensus.cc:1261] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb [term 2 FOLLOWER]: Refusing update from remote peer 694d2b1e07b84029a6232d1bbfba3be6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 14:08:59.668306 17490 consensus_queue.cc:1035] T b6e8031453c845e083866e9706373eb8 P 694d2b1e07b84029a6232d1bbfba3be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:08:59.668524 17491 consensus_queue.cc:1035] T b6e8031453c845e083866e9706373eb8 P 694d2b1e07b84029a6232d1bbfba3be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:08:59.670361 17490 raft_consensus.cc:2932] T b6e8031453c845e083866e9706373eb8 P 694d2b1e07b84029a6232d1bbfba3be6 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 547ad28f9bbf4ee2835750ec16680214 (127.16.13.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } attrs { replace: true } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } } peers { permanent_uuid: "547ad28f9bbf4ee2835750ec16680214" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 32965 } attrs { promote: true } } }
I20240501 14:08:59.670804 17260 raft_consensus.cc:2932] T b6e8031453c845e083866e9706373eb8 P 0333d5b1f6714097804614f828586e44 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 547ad28f9bbf4ee2835750ec16680214 (127.16.13.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } attrs { replace: true } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } } peers { permanent_uuid: "547ad28f9bbf4ee2835750ec16680214" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 32965 } attrs { promote: true } } }
W20240501 14:08:59.671415 17076 consensus_peers.cc:485] T b6e8031453c845e083866e9706373eb8 P 694d2b1e07b84029a6232d1bbfba3be6 -> Peer 547ad28f9bbf4ee2835750ec16680214 (127.16.13.196:32965): Couldn't send request to peer 547ad28f9bbf4ee2835750ec16680214. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6e8031453c845e083866e9706373eb8. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:08:59.671217 17186 raft_consensus.cc:2932] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 547ad28f9bbf4ee2835750ec16680214 (127.16.13.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } attrs { replace: true } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } } peers { permanent_uuid: "547ad28f9bbf4ee2835750ec16680214" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 32965 } attrs { promote: true } } }
I20240501 14:08:59.673280 17006 catalog_manager.cc:5484] T b6e8031453c845e083866e9706373eb8 P 0333d5b1f6714097804614f828586e44 reported cstate change: config changed from index -1 to 3, NON_VOTER 547ad28f9bbf4ee2835750ec16680214 (127.16.13.196) added. New cstate: current_term: 2 leader_uuid: "694d2b1e07b84029a6232d1bbfba3be6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "694d2b1e07b84029a6232d1bbfba3be6" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33287 } } peers { permanent_uuid: "a4a6589543cb487b81dd6f0ec0a85fcb" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 32803 } attrs { replace: true } } peers { permanent_uuid: "0333d5b1f6714097804614f828586e44" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 35511 } } peers { permanent_uuid: "547ad28f9bbf4ee2835750ec16680214" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 32965 } attrs { promote: true } } }
I20240501 14:08:59.700742 16439 tablet_server.cc:178] TabletServer@127.16.13.193:0 shutting down...
W20240501 14:08:59.701627 16994 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 1 similar messages]
W20240501 14:08:59.702610 17055 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 14:08:59.702695 17055 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240501 14:08:59.704528 16439 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:08:59.706756 16439 tablet_replica.cc:330] T f74eba74eed44278827bd37383ea632b P 694d2b1e07b84029a6232d1bbfba3be6: stopping tablet replica
I20240501 14:08:59.706902 16439 raft_consensus.cc:2229] T f74eba74eed44278827bd37383ea632b P 694d2b1e07b84029a6232d1bbfba3be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:08:59.707018 16439 raft_consensus.cc:2258] T f74eba74eed44278827bd37383ea632b P 694d2b1e07b84029a6232d1bbfba3be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:59.707455 16439 tablet_replica.cc:330] T 11957e1d5b084712b057d53625ec5e23 P 694d2b1e07b84029a6232d1bbfba3be6: stopping tablet replica
I20240501 14:08:59.707561 16439 raft_consensus.cc:2229] T 11957e1d5b084712b057d53625ec5e23 P 694d2b1e07b84029a6232d1bbfba3be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:08:59.707686 16439 raft_consensus.cc:2258] T 11957e1d5b084712b057d53625ec5e23 P 694d2b1e07b84029a6232d1bbfba3be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:59.708190 16439 tablet_replica.cc:330] T 86983f2828e143df8ef2aa8f61b46353 P 694d2b1e07b84029a6232d1bbfba3be6: stopping tablet replica
I20240501 14:08:59.708297 16439 raft_consensus.cc:2229] T 86983f2828e143df8ef2aa8f61b46353 P 694d2b1e07b84029a6232d1bbfba3be6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:08:59.708400 16439 raft_consensus.cc:2258] T 86983f2828e143df8ef2aa8f61b46353 P 694d2b1e07b84029a6232d1bbfba3be6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:59.708709 16439 tablet_replica.cc:330] T b6e8031453c845e083866e9706373eb8 P 694d2b1e07b84029a6232d1bbfba3be6: stopping tablet replica
I20240501 14:08:59.708806 16439 raft_consensus.cc:2229] T b6e8031453c845e083866e9706373eb8 P 694d2b1e07b84029a6232d1bbfba3be6 [term 2 LEADER]: Raft consensus shutting down.
I20240501 14:08:59.708988 16439 raft_consensus.cc:2258] T b6e8031453c845e083866e9706373eb8 P 694d2b1e07b84029a6232d1bbfba3be6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:59.709475 16439 tablet_replica.cc:330] T 08d8843520d54b7fb49be8b2cdead7b7 P 694d2b1e07b84029a6232d1bbfba3be6: stopping tablet replica
I20240501 14:08:59.709578 16439 raft_consensus.cc:2229] T 08d8843520d54b7fb49be8b2cdead7b7 P 694d2b1e07b84029a6232d1bbfba3be6 [term 2 LEADER]: Raft consensus shutting down.
I20240501 14:08:59.709723 16439 raft_consensus.cc:2258] T 08d8843520d54b7fb49be8b2cdead7b7 P 694d2b1e07b84029a6232d1bbfba3be6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:59.710076 16439 tablet_replica.cc:330] T 370d7c1dc64d40fa87489f566515bf30 P 694d2b1e07b84029a6232d1bbfba3be6: stopping tablet replica
I20240501 14:08:59.710144 16439 raft_consensus.cc:2229] T 370d7c1dc64d40fa87489f566515bf30 P 694d2b1e07b84029a6232d1bbfba3be6 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:08:59.710289 16439 raft_consensus.cc:2258] T 370d7c1dc64d40fa87489f566515bf30 P 694d2b1e07b84029a6232d1bbfba3be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:59.710587 16439 tablet_replica.cc:330] T eb2837529c914f5192cdaf3fae16afc1 P 694d2b1e07b84029a6232d1bbfba3be6: stopping tablet replica
I20240501 14:08:59.710692 16439 raft_consensus.cc:2229] T eb2837529c914f5192cdaf3fae16afc1 P 694d2b1e07b84029a6232d1bbfba3be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:08:59.710800 16439 raft_consensus.cc:2258] T eb2837529c914f5192cdaf3fae16afc1 P 694d2b1e07b84029a6232d1bbfba3be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:59.711239 16439 tablet_replica.cc:330] T e9b3a1494abb4116ae44a9711c535435 P 694d2b1e07b84029a6232d1bbfba3be6: stopping tablet replica
I20240501 14:08:59.711328 16439 raft_consensus.cc:2229] T e9b3a1494abb4116ae44a9711c535435 P 694d2b1e07b84029a6232d1bbfba3be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:08:59.711436 16439 raft_consensus.cc:2258] T e9b3a1494abb4116ae44a9711c535435 P 694d2b1e07b84029a6232d1bbfba3be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:59.715296 16439 tablet_server.cc:194] TabletServer@127.16.13.193:0 shutdown complete.
I20240501 14:08:59.717936 16439 tablet_server.cc:178] TabletServer@127.16.13.194:0 shutting down...
W20240501 14:08:59.720422 17055 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 14:08:59.720500 17055 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 14:08:59.722671 17055 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 14:08:59.722748 17055 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240501 14:08:59.722883 16439 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:08:59.723057 16439 tablet_replica.cc:330] T f74eba74eed44278827bd37383ea632b P a4a6589543cb487b81dd6f0ec0a85fcb: stopping tablet replica
I20240501 14:08:59.723233 16439 raft_consensus.cc:2229] T f74eba74eed44278827bd37383ea632b P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:08:59.723444 16439 raft_consensus.cc:2258] T f74eba74eed44278827bd37383ea632b P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:59.723907 16439 tablet_replica.cc:330] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb: stopping tablet replica
I20240501 14:08:59.724129 16439 raft_consensus.cc:2229] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:08:59.724342 16439 raft_consensus.cc:2258] T eb2837529c914f5192cdaf3fae16afc1 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:59.724821 16439 tablet_replica.cc:330] T e9b3a1494abb4116ae44a9711c535435 P a4a6589543cb487b81dd6f0ec0a85fcb: stopping tablet replica
I20240501 14:08:59.724933 16439 raft_consensus.cc:2229] T e9b3a1494abb4116ae44a9711c535435 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:08:59.725041 16439 raft_consensus.cc:2258] T e9b3a1494abb4116ae44a9711c535435 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:59.725543 16439 tablet_replica.cc:330] T 370d7c1dc64d40fa87489f566515bf30 P a4a6589543cb487b81dd6f0ec0a85fcb: stopping tablet replica
I20240501 14:08:59.725666 16439 raft_consensus.cc:2229] T 370d7c1dc64d40fa87489f566515bf30 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:08:59.725765 16439 raft_consensus.cc:2258] T 370d7c1dc64d40fa87489f566515bf30 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:59.726092 16439 tablet_replica.cc:330] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb: stopping tablet replica
I20240501 14:08:59.726215 16439 raft_consensus.cc:2229] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:08:59.726313 16439 raft_consensus.cc:2258] T 08d8843520d54b7fb49be8b2cdead7b7 P a4a6589543cb487b81dd6f0ec0a85fcb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:59.726636 16439 tablet_replica.cc:330] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb: stopping tablet replica
I20240501 14:08:59.726742 16439 raft_consensus.cc:2229] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:08:59.726850 16439 raft_consensus.cc:2258] T b6e8031453c845e083866e9706373eb8 P a4a6589543cb487b81dd6f0ec0a85fcb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:59.727403 16439 tablet_replica.cc:330] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb: stopping tablet replica
I20240501 14:08:59.727520 16439 raft_consensus.cc:2229] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:08:59.727699 16439 raft_consensus.cc:2258] T 11957e1d5b084712b057d53625ec5e23 P a4a6589543cb487b81dd6f0ec0a85fcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:59.728291 16439 tablet_replica.cc:330] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb: stopping tablet replica
I20240501 14:08:59.728394 16439 raft_consensus.cc:2229] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:08:59.728487 16439 raft_consensus.cc:2258] T 86983f2828e143df8ef2aa8f61b46353 P a4a6589543cb487b81dd6f0ec0a85fcb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:59.742470 16439 tablet_server.cc:194] TabletServer@127.16.13.194:0 shutdown complete.
I20240501 14:08:59.745361 16439 tablet_server.cc:178] TabletServer@127.16.13.195:0 shutting down...
W20240501 14:08:59.746460 17055 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 14:08:59.746575 17055 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240501 14:08:59.749920 16439 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:08:59.750079 16439 tablet_replica.cc:330] T b6e8031453c845e083866e9706373eb8 P 0333d5b1f6714097804614f828586e44: stopping tablet replica
I20240501 14:08:59.750211 16439 raft_consensus.cc:2229] T b6e8031453c845e083866e9706373eb8 P 0333d5b1f6714097804614f828586e44 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:08:59.750336 16439 raft_consensus.cc:2258] T b6e8031453c845e083866e9706373eb8 P 0333d5b1f6714097804614f828586e44 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:59.750890 16439 tablet_replica.cc:330] T 11957e1d5b084712b057d53625ec5e23 P 0333d5b1f6714097804614f828586e44: stopping tablet replica
I20240501 14:08:59.750999 16439 raft_consensus.cc:2229] T 11957e1d5b084712b057d53625ec5e23 P 0333d5b1f6714097804614f828586e44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:08:59.751103 16439 raft_consensus.cc:2258] T 11957e1d5b084712b057d53625ec5e23 P 0333d5b1f6714097804614f828586e44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:59.751570 16439 tablet_replica.cc:330] T 86983f2828e143df8ef2aa8f61b46353 P 0333d5b1f6714097804614f828586e44: stopping tablet replica
I20240501 14:08:59.751667 16439 raft_consensus.cc:2229] T 86983f2828e143df8ef2aa8f61b46353 P 0333d5b1f6714097804614f828586e44 [term 2 LEADER]: Raft consensus shutting down.
I20240501 14:08:59.751821 16439 raft_consensus.cc:2258] T 86983f2828e143df8ef2aa8f61b46353 P 0333d5b1f6714097804614f828586e44 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:59.752161 16439 tablet_replica.cc:330] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44: stopping tablet replica
I20240501 14:08:59.752265 16439 raft_consensus.cc:2229] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:08:59.752430 16439 raft_consensus.cc:2258] T e9b3a1494abb4116ae44a9711c535435 P 0333d5b1f6714097804614f828586e44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:59.752838 16439 tablet_replica.cc:330] T eb2837529c914f5192cdaf3fae16afc1 P 0333d5b1f6714097804614f828586e44: stopping tablet replica
I20240501 14:08:59.752934 16439 raft_consensus.cc:2229] T eb2837529c914f5192cdaf3fae16afc1 P 0333d5b1f6714097804614f828586e44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:08:59.753031 16439 raft_consensus.cc:2258] T eb2837529c914f5192cdaf3fae16afc1 P 0333d5b1f6714097804614f828586e44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:59.753415 16439 tablet_replica.cc:330] T 370d7c1dc64d40fa87489f566515bf30 P 0333d5b1f6714097804614f828586e44: stopping tablet replica
I20240501 14:08:59.753512 16439 raft_consensus.cc:2229] T 370d7c1dc64d40fa87489f566515bf30 P 0333d5b1f6714097804614f828586e44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:08:59.753602 16439 raft_consensus.cc:2258] T 370d7c1dc64d40fa87489f566515bf30 P 0333d5b1f6714097804614f828586e44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:59.753876 16439 tablet_replica.cc:330] T 08d8843520d54b7fb49be8b2cdead7b7 P 0333d5b1f6714097804614f828586e44: stopping tablet replica
I20240501 14:08:59.753966 16439 raft_consensus.cc:2229] T 08d8843520d54b7fb49be8b2cdead7b7 P 0333d5b1f6714097804614f828586e44 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:08:59.754052 16439 raft_consensus.cc:2258] T 08d8843520d54b7fb49be8b2cdead7b7 P 0333d5b1f6714097804614f828586e44 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:59.754330 16439 tablet_replica.cc:330] T f74eba74eed44278827bd37383ea632b P 0333d5b1f6714097804614f828586e44: stopping tablet replica
I20240501 14:08:59.754416 16439 raft_consensus.cc:2229] T f74eba74eed44278827bd37383ea632b P 0333d5b1f6714097804614f828586e44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:08:59.754504 16439 raft_consensus.cc:2258] T f74eba74eed44278827bd37383ea632b P 0333d5b1f6714097804614f828586e44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:59.767800 16439 tablet_server.cc:194] TabletServer@127.16.13.195:0 shutdown complete.
I20240501 14:08:59.770368 16439 tablet_server.cc:178] TabletServer@127.16.13.196:0 shutting down...
I20240501 14:08:59.775107 16439 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:08:59.787112 16439 tablet_server.cc:194] TabletServer@127.16.13.196:0 shutdown complete.
I20240501 14:08:59.789223 16439 master.cc:537] Master@127.16.13.254:42089 shutting down...
I20240501 14:08:59.793313 16439 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1aa9529348184ff1b8e0942957d9d84c [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:08:59.793478 16439 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1aa9529348184ff1b8e0942957d9d84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:59.793531 16439 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1aa9529348184ff1b8e0942957d9d84c: stopping tablet replica
I20240501 14:08:59.796176 16439 master.cc:552] Master@127.16.13.254:42089 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (7231 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20240501 14:08:59.807130 16439 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.13.254:33301,127.16.13.253:39083,127.16.13.252:42471
I20240501 14:08:59.807526 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:59.810326 17497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:59.811259 17500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:59.812260 17498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:59.813004 16439 server_base.cc:1034] running on GCE node
I20240501 14:08:59.813186 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:08:59.813225 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:08:59.813241 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572539813241 us; error 0 us; skew 500 ppm
I20240501 14:08:59.814172 16439 webserver.cc:460] Webserver started at http://127.16.13.254:40685/ using document root <none> and password file <none>
I20240501 14:08:59.814338 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:59.814400 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:59.814468 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:59.814878 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714572529860089-16439-0/minicluster-data/master-0-root/instance:
uuid: "f76fcf8e69df45e39d4c4f51fb73be56"
format_stamp: "Formatted at 2024-05-01 14:08:59 on dist-test-slave-mgxc"
I20240501 14:08:59.816476 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 14:08:59.817453 17505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:59.817806 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:59.817886 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714572529860089-16439-0/minicluster-data/master-0-root
uuid: "f76fcf8e69df45e39d4c4f51fb73be56"
format_stamp: "Formatted at 2024-05-01 14:08:59 on dist-test-slave-mgxc"
I20240501 14:08:59.817948 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714572529860089-16439-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714572529860089-16439-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714572529860089-16439-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:59.826555 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:59.826859 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:59.830513 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.254:33301
I20240501 14:08:59.831576 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 14:08:59.836055 17558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:59.836074 17557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.254:33301 every 8 connection(s)
I20240501 14:08:59.837539 17558 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.13.254" port: 33301 } has no permanent_uuid. Determining permanent_uuid...
W20240501 14:08:59.838028 17560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:59.838732 17561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:59.839756 17563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:59.840548 16439 server_base.cc:1034] running on GCE node
I20240501 14:08:59.841010 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:08:59.841069 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:08:59.841097 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572539841096 us; error 0 us; skew 500 ppm
I20240501 14:08:59.842077 16439 webserver.cc:460] Webserver started at http://127.16.13.253:36851/ using document root <none> and password file <none>
I20240501 14:08:59.842159 17558 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.13.253" port: 39083 } has no permanent_uuid. Determining permanent_uuid...
I20240501 14:08:59.842228 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:59.842272 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:59.842345 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:59.842803 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714572529860089-16439-0/minicluster-data/master-1-root/instance:
uuid: "33806aa6a42e403e84358aa7fe3e63c7"
format_stamp: "Formatted at 2024-05-01 14:08:59 on dist-test-slave-mgxc"
W20240501 14:08:59.843466 17558 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.16.13.253:39083: Network error: Client connection negotiation failed: client connection to 127.16.13.253:39083: connect: Connection refused (error 111)
I20240501 14:08:59.844411 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 14:08:59.845373 17571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:59.845652 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 14:08:59.845713 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714572529860089-16439-0/minicluster-data/master-1-root
uuid: "33806aa6a42e403e84358aa7fe3e63c7"
format_stamp: "Formatted at 2024-05-01 14:08:59 on dist-test-slave-mgxc"
I20240501 14:08:59.845777 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714572529860089-16439-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714572529860089-16439-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714572529860089-16439-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:59.856776 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:59.857127 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:59.860900 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.253:39083
I20240501 14:08:59.860929 17623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.253:39083 every 8 connection(s)
I20240501 14:08:59.861562 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 14:08:59.861757 17624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:59.863210 17624 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.13.254" port: 33301 } has no permanent_uuid. Determining permanent_uuid...
W20240501 14:08:59.864240 17627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:59.864902 17626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:59.865939 17629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:59.866778 16439 server_base.cc:1034] running on GCE node
I20240501 14:08:59.866978 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:08:59.867029 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:08:59.867054 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572539867053 us; error 0 us; skew 500 ppm
I20240501 14:08:59.867628 17624 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.13.253" port: 39083 } has no permanent_uuid. Determining permanent_uuid...
I20240501 14:08:59.868075 16439 webserver.cc:460] Webserver started at http://127.16.13.252:37539/ using document root <none> and password file <none>
I20240501 14:08:59.868237 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:59.868296 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:59.868439 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:59.868893 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714572529860089-16439-0/minicluster-data/master-2-root/instance:
uuid: "b34a588c4a864adca592f5081c252e99"
format_stamp: "Formatted at 2024-05-01 14:08:59 on dist-test-slave-mgxc"
I20240501 14:08:59.870472 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 14:08:59.871665 17636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:59.871949 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 14:08:59.872023 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714572529860089-16439-0/minicluster-data/master-2-root
uuid: "b34a588c4a864adca592f5081c252e99"
format_stamp: "Formatted at 2024-05-01 14:08:59 on dist-test-slave-mgxc"
I20240501 14:08:59.872093 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714572529860089-16439-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714572529860089-16439-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714572529860089-16439-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:59.872169 17624 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.13.252" port: 42471 } has no permanent_uuid. Determining permanent_uuid...
W20240501 14:08:59.873346 17624 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.16.13.252:42471: Network error: Client connection negotiation failed: client connection to 127.16.13.252:42471: connect: Connection refused (error 111)
I20240501 14:08:59.877631 17558 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.13.253" port: 39083 } attempt: 1
I20240501 14:08:59.881443 17558 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.13.252" port: 42471 } has no permanent_uuid. Determining permanent_uuid...
W20240501 14:08:59.882546 17558 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.16.13.252:42471: Network error: Client connection negotiation failed: client connection to 127.16.13.252:42471: connect: Connection refused (error 111)
I20240501 14:08:59.890062 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:59.890419 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:59.894651 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.252:42471
I20240501 14:08:59.894692 17689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.252:42471 every 8 connection(s)
I20240501 14:08:59.895078 16439 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 14:08:59.895506 17690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:59.896700 17690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.13.254" port: 33301 } has no permanent_uuid. Determining permanent_uuid...
I20240501 14:08:59.900755 17690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.13.253" port: 39083 } has no permanent_uuid. Determining permanent_uuid...
I20240501 14:08:59.904287 17690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.13.252" port: 42471 } has no permanent_uuid. Determining permanent_uuid...
I20240501 14:08:59.909016 17690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b34a588c4a864adca592f5081c252e99: Bootstrap starting.
I20240501 14:08:59.909822 17690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b34a588c4a864adca592f5081c252e99: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:59.910785 17690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b34a588c4a864adca592f5081c252e99: No bootstrap required, opened a new log
I20240501 14:08:59.911147 17690 raft_consensus.cc:348] T 00000000000000000000000000000000 P b34a588c4a864adca592f5081c252e99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76fcf8e69df45e39d4c4f51fb73be56" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 33301 } } peers { permanent_uuid: "33806aa6a42e403e84358aa7fe3e63c7" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 39083 } } peers { permanent_uuid: "b34a588c4a864adca592f5081c252e99" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 42471 } }
I20240501 14:08:59.911242 17690 raft_consensus.cc:374] T 00000000000000000000000000000000 P b34a588c4a864adca592f5081c252e99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:59.911264 17690 raft_consensus.cc:729] T 00000000000000000000000000000000 P b34a588c4a864adca592f5081c252e99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b34a588c4a864adca592f5081c252e99, State: Initialized, Role: FOLLOWER
I20240501 14:08:59.911399 17690 consensus_queue.cc:260] T 00000000000000000000000000000000 P b34a588c4a864adca592f5081c252e99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76fcf8e69df45e39d4c4f51fb73be56" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 33301 } } peers { permanent_uuid: "33806aa6a42e403e84358aa7fe3e63c7" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 39083 } } peers { permanent_uuid: "b34a588c4a864adca592f5081c252e99" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 42471 } }
I20240501 14:08:59.911684 17695 sys_catalog.cc:455] T 00000000000000000000000000000000 P b34a588c4a864adca592f5081c252e99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76fcf8e69df45e39d4c4f51fb73be56" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 33301 } } peers { permanent_uuid: "33806aa6a42e403e84358aa7fe3e63c7" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 39083 } } peers { permanent_uuid: "b34a588c4a864adca592f5081c252e99" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 42471 } } }
I20240501 14:08:59.911777 17695 sys_catalog.cc:458] T 00000000000000000000000000000000 P b34a588c4a864adca592f5081c252e99 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:08:59.911808 17690 sys_catalog.cc:564] T 00000000000000000000000000000000 P b34a588c4a864adca592f5081c252e99 [sys.catalog]: configured and running, proceeding with master startup.
W20240501 14:08:59.914041 17706 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b34a588c4a864adca592f5081c252e99: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 14:08:59.914105 17706 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 14:08:59.927515 17624 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.13.252" port: 42471 } attempt: 1
I20240501 14:08:59.932775 17624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7: Bootstrap starting.
I20240501 14:08:59.933647 17624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:59.934835 17624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7: No bootstrap required, opened a new log
I20240501 14:08:59.935292 17624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76fcf8e69df45e39d4c4f51fb73be56" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 33301 } } peers { permanent_uuid: "33806aa6a42e403e84358aa7fe3e63c7" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 39083 } } peers { permanent_uuid: "b34a588c4a864adca592f5081c252e99" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 42471 } }
I20240501 14:08:59.935392 17624 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:59.935417 17624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33806aa6a42e403e84358aa7fe3e63c7, State: Initialized, Role: FOLLOWER
I20240501 14:08:59.935537 17624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76fcf8e69df45e39d4c4f51fb73be56" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 33301 } } peers { permanent_uuid: "33806aa6a42e403e84358aa7fe3e63c7" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 39083 } } peers { permanent_uuid: "b34a588c4a864adca592f5081c252e99" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 42471 } }
I20240501 14:08:59.936023 17624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:08:59.935895 17708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76fcf8e69df45e39d4c4f51fb73be56" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 33301 } } peers { permanent_uuid: "33806aa6a42e403e84358aa7fe3e63c7" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 39083 } } peers { permanent_uuid: "b34a588c4a864adca592f5081c252e99" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 42471 } } }
I20240501 14:08:59.936141 17708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7 [sys.catalog]: This master's current role is: FOLLOWER
W20240501 14:08:59.938336 17719 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 14:08:59.938408 17719 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 14:08:59.942723 17558 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.13.252" port: 42471 } attempt: 1
I20240501 14:08:59.947711 17558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56: Bootstrap starting.
I20240501 14:08:59.948657 17558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:59.949815 17558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56: No bootstrap required, opened a new log
I20240501 14:08:59.950282 17558 raft_consensus.cc:348] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76fcf8e69df45e39d4c4f51fb73be56" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 33301 } } peers { permanent_uuid: "33806aa6a42e403e84358aa7fe3e63c7" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 39083 } } peers { permanent_uuid: "b34a588c4a864adca592f5081c252e99" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 42471 } }
I20240501 14:08:59.950402 17558 raft_consensus.cc:374] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:59.950429 17558 raft_consensus.cc:729] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f76fcf8e69df45e39d4c4f51fb73be56, State: Initialized, Role: FOLLOWER
I20240501 14:08:59.950534 17558 consensus_queue.cc:260] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76fcf8e69df45e39d4c4f51fb73be56" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 33301 } } peers { permanent_uuid: "33806aa6a42e403e84358aa7fe3e63c7" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 39083 } } peers { permanent_uuid: "b34a588c4a864adca592f5081c252e99" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 42471 } }
I20240501 14:08:59.950840 17721 sys_catalog.cc:455] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76fcf8e69df45e39d4c4f51fb73be56" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 33301 } } peers { permanent_uuid: "33806aa6a42e403e84358aa7fe3e63c7" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 39083 } } peers { permanent_uuid: "b34a588c4a864adca592f5081c252e99" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 42471 } } }
I20240501 14:08:59.950935 17721 sys_catalog.cc:458] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:08:59.950944 17558 sys_catalog.cc:564] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:08:59.951086 17708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:59.951179 17708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76fcf8e69df45e39d4c4f51fb73be56" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 33301 } } peers { permanent_uuid: "33806aa6a42e403e84358aa7fe3e63c7" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 39083 } } peers { permanent_uuid: "b34a588c4a864adca592f5081c252e99" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 42471 } }
I20240501 14:08:59.951546 17708 leader_election.cc:288] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f76fcf8e69df45e39d4c4f51fb73be56 (127.16.13.254:33301), b34a588c4a864adca592f5081c252e99 (127.16.13.252:42471)
I20240501 14:08:59.951938 17532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "33806aa6a42e403e84358aa7fe3e63c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f76fcf8e69df45e39d4c4f51fb73be56" is_pre_election: true
I20240501 14:08:59.951946 17664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "33806aa6a42e403e84358aa7fe3e63c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b34a588c4a864adca592f5081c252e99" is_pre_election: true
I20240501 14:08:59.952101 17532 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33806aa6a42e403e84358aa7fe3e63c7 in term 0.
I20240501 14:08:59.952105 17664 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b34a588c4a864adca592f5081c252e99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33806aa6a42e403e84358aa7fe3e63c7 in term 0.
I20240501 14:08:59.952417 17575 leader_election.cc:302] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33806aa6a42e403e84358aa7fe3e63c7, b34a588c4a864adca592f5081c252e99; no voters:
I20240501 14:08:59.952561 17708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:59.952613 17708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:59.952654 17708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:59.953135 16439 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240501 14:08:59.953195 16439 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240501 14:08:59.953538 17708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76fcf8e69df45e39d4c4f51fb73be56" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 33301 } } peers { permanent_uuid: "33806aa6a42e403e84358aa7fe3e63c7" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 39083 } } peers { permanent_uuid: "b34a588c4a864adca592f5081c252e99" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 42471 } }
W20240501 14:08:59.953661 17732 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20240501 14:08:59.953666 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:59.953747 17732 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 14:08:59.953910 17708 leader_election.cc:288] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7 [CANDIDATE]: Term 1 election: Requested vote from peers f76fcf8e69df45e39d4c4f51fb73be56 (127.16.13.254:33301), b34a588c4a864adca592f5081c252e99 (127.16.13.252:42471)
I20240501 14:08:59.954154 17532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "33806aa6a42e403e84358aa7fe3e63c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f76fcf8e69df45e39d4c4f51fb73be56"
I20240501 14:08:59.954252 17532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:59.955269 17532 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33806aa6a42e403e84358aa7fe3e63c7 in term 1.
I20240501 14:08:59.955467 17664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "33806aa6a42e403e84358aa7fe3e63c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b34a588c4a864adca592f5081c252e99"
I20240501 14:08:59.955574 17664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b34a588c4a864adca592f5081c252e99 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:59.955579 17574 leader_election.cc:302] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33806aa6a42e403e84358aa7fe3e63c7, f76fcf8e69df45e39d4c4f51fb73be56; no voters:
I20240501 14:08:59.955811 17708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:59.955971 17708 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7 [term 1 LEADER]: Becoming Leader. State: Replica: 33806aa6a42e403e84358aa7fe3e63c7, State: Running, Role: LEADER
I20240501 14:08:59.956094 17708 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76fcf8e69df45e39d4c4f51fb73be56" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 33301 } } peers { permanent_uuid: "33806aa6a42e403e84358aa7fe3e63c7" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 39083 } } peers { permanent_uuid: "b34a588c4a864adca592f5081c252e99" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 42471 } }
I20240501 14:08:59.956527 17664 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b34a588c4a864adca592f5081c252e99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33806aa6a42e403e84358aa7fe3e63c7 in term 1.
W20240501 14:08:59.956595 17733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:59.956818 17738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33806aa6a42e403e84358aa7fe3e63c7. Latest consensus state: current_term: 1 leader_uuid: "33806aa6a42e403e84358aa7fe3e63c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76fcf8e69df45e39d4c4f51fb73be56" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 33301 } } peers { permanent_uuid: "33806aa6a42e403e84358aa7fe3e63c7" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 39083 } } peers { permanent_uuid: "b34a588c4a864adca592f5081c252e99" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 42471 } } }
I20240501 14:08:59.957041 17738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7 [sys.catalog]: This master's current role is: LEADER
I20240501 14:08:59.957417 17740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20240501 14:08:59.957545 17736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:59.958115 17740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20240501 14:08:59.958302 17734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:59.959357 16439 server_base.cc:1034] running on GCE node
I20240501 14:08:59.959517 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:08:59.959554 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:08:59.959570 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572539959570 us; error 0 us; skew 500 ppm
I20240501 14:08:59.959874 17664 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b34a588c4a864adca592f5081c252e99 [term 1 FOLLOWER]: Refusing update from remote peer 33806aa6a42e403e84358aa7fe3e63c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:59.959996 17532 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [term 1 FOLLOWER]: Refusing update from remote peer 33806aa6a42e403e84358aa7fe3e63c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:59.960391 17738 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b34a588c4a864adca592f5081c252e99" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 42471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:59.960645 17708 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f76fcf8e69df45e39d4c4f51fb73be56" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 33301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:59.960731 16439 webserver.cc:460] Webserver started at http://127.16.13.193:45859/ using document root <none> and password file <none>
I20240501 14:08:59.960887 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:59.960947 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:59.961017 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:59.961395 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714572529860089-16439-0/minicluster-data/ts-0-root/instance:
uuid: "9b95286c84c84e99aefecb5d5a5fb345"
format_stamp: "Formatted at 2024-05-01 14:08:59 on dist-test-slave-mgxc"
I20240501 14:08:59.961803 17695 sys_catalog.cc:455] T 00000000000000000000000000000000 P b34a588c4a864adca592f5081c252e99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33806aa6a42e403e84358aa7fe3e63c7. Latest consensus state: current_term: 1 leader_uuid: "33806aa6a42e403e84358aa7fe3e63c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76fcf8e69df45e39d4c4f51fb73be56" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 33301 } } peers { permanent_uuid: "33806aa6a42e403e84358aa7fe3e63c7" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 39083 } } peers { permanent_uuid: "b34a588c4a864adca592f5081c252e99" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 42471 } } }
I20240501 14:08:59.961898 17695 sys_catalog.cc:458] T 00000000000000000000000000000000 P b34a588c4a864adca592f5081c252e99 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:08:59.961881 17721 sys_catalog.cc:455] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33806aa6a42e403e84358aa7fe3e63c7. Latest consensus state: current_term: 1 leader_uuid: "33806aa6a42e403e84358aa7fe3e63c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76fcf8e69df45e39d4c4f51fb73be56" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 33301 } } peers { permanent_uuid: "33806aa6a42e403e84358aa7fe3e63c7" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 39083 } } peers { permanent_uuid: "b34a588c4a864adca592f5081c252e99" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 42471 } } }
I20240501 14:08:59.961952 17721 sys_catalog.cc:458] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:08:59.962944 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:59.963421 17738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "33806aa6a42e403e84358aa7fe3e63c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76fcf8e69df45e39d4c4f51fb73be56" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 33301 } } peers { permanent_uuid: "33806aa6a42e403e84358aa7fe3e63c7" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 39083 } } peers { permanent_uuid: "b34a588c4a864adca592f5081c252e99" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 42471 } } }
I20240501 14:08:59.963531 17738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7 [sys.catalog]: This master's current role is: LEADER
I20240501 14:08:59.963634 17738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "33806aa6a42e403e84358aa7fe3e63c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76fcf8e69df45e39d4c4f51fb73be56" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 33301 } } peers { permanent_uuid: "33806aa6a42e403e84358aa7fe3e63c7" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 39083 } } peers { permanent_uuid: "b34a588c4a864adca592f5081c252e99" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 42471 } } }
I20240501 14:08:59.963703 17738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7 [sys.catalog]: This master's current role is: LEADER
I20240501 14:08:59.963966 17695 sys_catalog.cc:455] T 00000000000000000000000000000000 P b34a588c4a864adca592f5081c252e99 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33806aa6a42e403e84358aa7fe3e63c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76fcf8e69df45e39d4c4f51fb73be56" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 33301 } } peers { permanent_uuid: "33806aa6a42e403e84358aa7fe3e63c7" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 39083 } } peers { permanent_uuid: "b34a588c4a864adca592f5081c252e99" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 42471 } } }
I20240501 14:08:59.964067 17695 sys_catalog.cc:458] T 00000000000000000000000000000000 P b34a588c4a864adca592f5081c252e99 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:08:59.964361 17740 catalog_manager.cc:1347] Generated new cluster ID: 432f4f7750ab41ea97beb1f10fe613b5
I20240501 14:08:59.964470 17740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:08:59.968205 17752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:59.968479 17721 sys_catalog.cc:455] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33806aa6a42e403e84358aa7fe3e63c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76fcf8e69df45e39d4c4f51fb73be56" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 33301 } } peers { permanent_uuid: "33806aa6a42e403e84358aa7fe3e63c7" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 39083 } } peers { permanent_uuid: "b34a588c4a864adca592f5081c252e99" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 42471 } } }
I20240501 14:08:59.968582 17721 sys_catalog.cc:458] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:08:59.968621 16439 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20240501 14:08:59.968676 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714572529860089-16439-0/minicluster-data/ts-0-root
uuid: "9b95286c84c84e99aefecb5d5a5fb345"
format_stamp: "Formatted at 2024-05-01 14:08:59 on dist-test-slave-mgxc"
I20240501 14:08:59.968734 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714572529860089-16439-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714572529860089-16439-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714572529860089-16439-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:59.973512 17740 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:08:59.974238 17740 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:08:59.984388 17740 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7: Generated new TSK 0
I20240501 14:08:59.984606 17740 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:08:59.987421 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:59.987787 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:59.988075 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:59.988552 16439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:59.988588 16439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:59.988624 16439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:59.988654 16439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:59.993037 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.193:33927
I20240501 14:08:59.993098 17816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.193:33927 every 8 connection(s)
I20240501 14:08:59.994019 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:59.997434 17825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:59.998456 17827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:59.998750 17819 heartbeater.cc:347] Connected to a master server at 127.16.13.253:39083
I20240501 14:08:59.998847 17819 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:59.998888 17818 heartbeater.cc:347] Connected to a master server at 127.16.13.254:33301
I20240501 14:08:59.998935 17818 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:59.999045 17817 heartbeater.cc:347] Connected to a master server at 127.16.13.252:42471
I20240501 14:08:59.999065 17818 heartbeater.cc:511] Master 127.16.13.254:33301 requested a full tablet report, sending...
I20240501 14:08:59.999130 17817 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:59.999135 17819 heartbeater.cc:511] Master 127.16.13.253:39083 requested a full tablet report, sending...
W20240501 14:08:59.999280 17824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:59.999395 17817 heartbeater.cc:511] Master 127.16.13.252:42471 requested a full tablet report, sending...
I20240501 14:08:59.999775 17522 ts_manager.cc:194] Registered new tserver with Master: 9b95286c84c84e99aefecb5d5a5fb345 (127.16.13.193:33927)
I20240501 14:08:59.999902 17588 ts_manager.cc:194] Registered new tserver with Master: 9b95286c84c84e99aefecb5d5a5fb345 (127.16.13.193:33927)
I20240501 14:08:59.999995 17654 ts_manager.cc:194] Registered new tserver with Master: 9b95286c84c84e99aefecb5d5a5fb345 (127.16.13.193:33927)
I20240501 14:09:00.000684 17588 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59960
I20240501 14:09:00.001021 16439 server_base.cc:1034] running on GCE node
I20240501 14:09:00.001207 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:09:00.001255 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:09:00.001300 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572540001299 us; error 0 us; skew 500 ppm
I20240501 14:09:00.002307 16439 webserver.cc:460] Webserver started at http://127.16.13.194:43975/ using document root <none> and password file <none>
I20240501 14:09:00.002468 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:00.002514 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:00.002591 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:00.002966 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714572529860089-16439-0/minicluster-data/ts-1-root/instance:
uuid: "31663ec98b49478b84f80767cfe3f33e"
format_stamp: "Formatted at 2024-05-01 14:08:59 on dist-test-slave-mgxc"
I20240501 14:09:00.004366 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 14:09:00.005224 17832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:00.005426 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:00.005492 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714572529860089-16439-0/minicluster-data/ts-1-root
uuid: "31663ec98b49478b84f80767cfe3f33e"
format_stamp: "Formatted at 2024-05-01 14:08:59 on dist-test-slave-mgxc"
I20240501 14:09:00.005554 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714572529860089-16439-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714572529860089-16439-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714572529860089-16439-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:00.017752 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:00.018188 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:00.018469 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:00.018913 16439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:00.018954 16439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:00.018998 16439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:00.019028 16439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:00.023389 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.194:44673
I20240501 14:09:00.023487 17895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.194:44673 every 8 connection(s)
I20240501 14:09:00.024346 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:00.028278 17903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:00.029266 17897 heartbeater.cc:347] Connected to a master server at 127.16.13.254:33301
I20240501 14:09:00.029377 17897 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:00.029573 17898 heartbeater.cc:347] Connected to a master server at 127.16.13.253:39083
W20240501 14:09:00.029345 17906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:00.029646 17897 heartbeater.cc:511] Master 127.16.13.254:33301 requested a full tablet report, sending...
I20240501 14:09:00.029646 17898 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:00.029644 17896 heartbeater.cc:347] Connected to a master server at 127.16.13.252:42471
I20240501 14:09:00.029840 17896 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:00.029981 17898 heartbeater.cc:511] Master 127.16.13.253:39083 requested a full tablet report, sending...
I20240501 14:09:00.030027 17896 heartbeater.cc:511] Master 127.16.13.252:42471 requested a full tablet report, sending...
W20240501 14:09:00.030428 17904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:00.030475 17522 ts_manager.cc:194] Registered new tserver with Master: 31663ec98b49478b84f80767cfe3f33e (127.16.13.194:44673)
I20240501 14:09:00.030541 17588 ts_manager.cc:194] Registered new tserver with Master: 31663ec98b49478b84f80767cfe3f33e (127.16.13.194:44673)
I20240501 14:09:00.030578 17654 ts_manager.cc:194] Registered new tserver with Master: 31663ec98b49478b84f80767cfe3f33e (127.16.13.194:44673)
I20240501 14:09:00.030905 16439 server_base.cc:1034] running on GCE node
I20240501 14:09:00.031108 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:09:00.031153 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:09:00.031181 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572540031180 us; error 0 us; skew 500 ppm
I20240501 14:09:00.031272 17588 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59974
I20240501 14:09:00.032353 16439 webserver.cc:460] Webserver started at http://127.16.13.195:46749/ using document root <none> and password file <none>
I20240501 14:09:00.032511 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:00.032558 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:00.032631 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:00.032979 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714572529860089-16439-0/minicluster-data/ts-2-root/instance:
uuid: "3ef63aea111c483faa410fa692f3eff2"
format_stamp: "Formatted at 2024-05-01 14:09:00 on dist-test-slave-mgxc"
I20240501 14:09:00.034358 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:00.035189 17911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:00.035423 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:00.035488 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714572529860089-16439-0/minicluster-data/ts-2-root
uuid: "3ef63aea111c483faa410fa692f3eff2"
format_stamp: "Formatted at 2024-05-01 14:09:00 on dist-test-slave-mgxc"
I20240501 14:09:00.035550 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714572529860089-16439-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714572529860089-16439-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714572529860089-16439-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:00.053818 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:00.054225 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:00.054512 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:00.054963 16439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:00.055003 16439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:00.055047 16439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:00.055081 16439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:00.059365 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.195:38489
I20240501 14:09:00.059492 17974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.195:38489 every 8 connection(s)
I20240501 14:09:00.064747 17976 heartbeater.cc:347] Connected to a master server at 127.16.13.254:33301
I20240501 14:09:00.064858 17977 heartbeater.cc:347] Connected to a master server at 127.16.13.253:39083
I20240501 14:09:00.064880 17976 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:00.064958 17977 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:00.065142 17977 heartbeater.cc:511] Master 127.16.13.253:39083 requested a full tablet report, sending...
I20240501 14:09:00.065148 17976 heartbeater.cc:511] Master 127.16.13.254:33301 requested a full tablet report, sending...
I20240501 14:09:00.065171 17975 heartbeater.cc:347] Connected to a master server at 127.16.13.252:42471
I20240501 14:09:00.065299 17975 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:00.065575 17975 heartbeater.cc:511] Master 127.16.13.252:42471 requested a full tablet report, sending...
I20240501 14:09:00.065763 17588 ts_manager.cc:194] Registered new tserver with Master: 3ef63aea111c483faa410fa692f3eff2 (127.16.13.195:38489)
I20240501 14:09:00.065800 17522 ts_manager.cc:194] Registered new tserver with Master: 3ef63aea111c483faa410fa692f3eff2 (127.16.13.195:38489)
I20240501 14:09:00.066161 17654 ts_manager.cc:194] Registered new tserver with Master: 3ef63aea111c483faa410fa692f3eff2 (127.16.13.195:38489)
I20240501 14:09:00.066377 17588 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59986
I20240501 14:09:00.067050 16439 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.00713068s
I20240501 14:09:00.914896 17706 catalog_manager.cc:1259] Loaded cluster ID: 432f4f7750ab41ea97beb1f10fe613b5
I20240501 14:09:00.914985 17706 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b34a588c4a864adca592f5081c252e99: loading cluster ID for follower catalog manager: success
I20240501 14:09:00.916154 17706 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b34a588c4a864adca592f5081c252e99: acquiring CA information for follower catalog manager: success
I20240501 14:09:00.916687 17706 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b34a588c4a864adca592f5081c252e99: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 14:09:00.954571 17732 catalog_manager.cc:1259] Loaded cluster ID: 432f4f7750ab41ea97beb1f10fe613b5
I20240501 14:09:00.954671 17732 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56: loading cluster ID for follower catalog manager: success
I20240501 14:09:00.955852 17732 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56: acquiring CA information for follower catalog manager: success
I20240501 14:09:00.956405 17732 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 14:09:01.001870 17819 heartbeater.cc:503] Master 127.16.13.253:39083 was elected leader, sending a full tablet report...
I20240501 14:09:01.032260 17898 heartbeater.cc:503] Master 127.16.13.253:39083 was elected leader, sending a full tablet report...
I20240501 14:09:01.067066 17977 heartbeater.cc:503] Master 127.16.13.253:39083 was elected leader, sending a full tablet report...
I20240501 14:09:01.937963 17718 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:09:02.126577 16439 test_util.cc:268] Using random seed: 122726895
I20240501 14:09:02.137030 17588 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50236:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:09:02.153772 17939 tablet_service.cc:1467] Processing CreateTablet for tablet 33d83caedf904c7ba1bdbf9e5a0aa7a9 (DEFAULT_TABLE table=test-workload [id=ff9b6cac3ae949268f57f34724632176]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240501 14:09:02.154083 17939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33d83caedf904c7ba1bdbf9e5a0aa7a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:02.154884 17860 tablet_service.cc:1467] Processing CreateTablet for tablet a5245af584e84fd295008a34a804da1c (DEFAULT_TABLE table=test-workload [id=ff9b6cac3ae949268f57f34724632176]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240501 14:09:02.155157 17860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5245af584e84fd295008a34a804da1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:02.156312 18010 tablet_bootstrap.cc:492] T 33d83caedf904c7ba1bdbf9e5a0aa7a9 P 3ef63aea111c483faa410fa692f3eff2: Bootstrap starting.
I20240501 14:09:02.157290 18010 tablet_bootstrap.cc:654] T 33d83caedf904c7ba1bdbf9e5a0aa7a9 P 3ef63aea111c483faa410fa692f3eff2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:02.158181 18011 tablet_bootstrap.cc:492] T a5245af584e84fd295008a34a804da1c P 31663ec98b49478b84f80767cfe3f33e: Bootstrap starting.
I20240501 14:09:02.158200 17781 tablet_service.cc:1467] Processing CreateTablet for tablet 42fe800961734151bb412c007d1d5b99 (DEFAULT_TABLE table=test-workload [id=ff9b6cac3ae949268f57f34724632176]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240501 14:09:02.158488 17781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42fe800961734151bb412c007d1d5b99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:02.159336 18011 tablet_bootstrap.cc:654] T a5245af584e84fd295008a34a804da1c P 31663ec98b49478b84f80767cfe3f33e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:02.160251 18014 tablet_bootstrap.cc:492] T 42fe800961734151bb412c007d1d5b99 P 9b95286c84c84e99aefecb5d5a5fb345: Bootstrap starting.
I20240501 14:09:02.160586 18010 tablet_bootstrap.cc:492] T 33d83caedf904c7ba1bdbf9e5a0aa7a9 P 3ef63aea111c483faa410fa692f3eff2: No bootstrap required, opened a new log
I20240501 14:09:02.160737 18010 ts_tablet_manager.cc:1367] T 33d83caedf904c7ba1bdbf9e5a0aa7a9 P 3ef63aea111c483faa410fa692f3eff2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:02.161156 18014 tablet_bootstrap.cc:654] T 42fe800961734151bb412c007d1d5b99 P 9b95286c84c84e99aefecb5d5a5fb345: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:02.161326 18010 raft_consensus.cc:348] T 33d83caedf904c7ba1bdbf9e5a0aa7a9 P 3ef63aea111c483faa410fa692f3eff2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ef63aea111c483faa410fa692f3eff2" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 38489 } }
I20240501 14:09:02.161449 18010 raft_consensus.cc:374] T 33d83caedf904c7ba1bdbf9e5a0aa7a9 P 3ef63aea111c483faa410fa692f3eff2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:02.161545 18010 raft_consensus.cc:729] T 33d83caedf904c7ba1bdbf9e5a0aa7a9 P 3ef63aea111c483faa410fa692f3eff2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ef63aea111c483faa410fa692f3eff2, State: Initialized, Role: FOLLOWER
I20240501 14:09:02.161720 18010 consensus_queue.cc:260] T 33d83caedf904c7ba1bdbf9e5a0aa7a9 P 3ef63aea111c483faa410fa692f3eff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ef63aea111c483faa410fa692f3eff2" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 38489 } }
I20240501 14:09:02.161834 18010 raft_consensus.cc:388] T 33d83caedf904c7ba1bdbf9e5a0aa7a9 P 3ef63aea111c483faa410fa692f3eff2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:09:02.161883 18010 raft_consensus.cc:482] T 33d83caedf904c7ba1bdbf9e5a0aa7a9 P 3ef63aea111c483faa410fa692f3eff2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:09:02.161945 18010 raft_consensus.cc:3037] T 33d83caedf904c7ba1bdbf9e5a0aa7a9 P 3ef63aea111c483faa410fa692f3eff2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:02.163041 18010 raft_consensus.cc:504] T 33d83caedf904c7ba1bdbf9e5a0aa7a9 P 3ef63aea111c483faa410fa692f3eff2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ef63aea111c483faa410fa692f3eff2" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 38489 } }
I20240501 14:09:02.163218 18010 leader_election.cc:302] T 33d83caedf904c7ba1bdbf9e5a0aa7a9 P 3ef63aea111c483faa410fa692f3eff2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ef63aea111c483faa410fa692f3eff2; no voters:
I20240501 14:09:02.163466 18010 leader_election.cc:288] T 33d83caedf904c7ba1bdbf9e5a0aa7a9 P 3ef63aea111c483faa410fa692f3eff2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:09:02.163563 18016 raft_consensus.cc:2781] T 33d83caedf904c7ba1bdbf9e5a0aa7a9 P 3ef63aea111c483faa410fa692f3eff2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:02.163797 18016 raft_consensus.cc:686] T 33d83caedf904c7ba1bdbf9e5a0aa7a9 P 3ef63aea111c483faa410fa692f3eff2 [term 1 LEADER]: Becoming Leader. State: Replica: 3ef63aea111c483faa410fa692f3eff2, State: Running, Role: LEADER
I20240501 14:09:02.164305 18016 consensus_queue.cc:237] T 33d83caedf904c7ba1bdbf9e5a0aa7a9 P 3ef63aea111c483faa410fa692f3eff2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ef63aea111c483faa410fa692f3eff2" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 38489 } }
I20240501 14:09:02.164594 18011 tablet_bootstrap.cc:492] T a5245af584e84fd295008a34a804da1c P 31663ec98b49478b84f80767cfe3f33e: No bootstrap required, opened a new log
I20240501 14:09:02.164748 18011 ts_tablet_manager.cc:1367] T a5245af584e84fd295008a34a804da1c P 31663ec98b49478b84f80767cfe3f33e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240501 14:09:02.163898 18010 ts_tablet_manager.cc:1398] T 33d83caedf904c7ba1bdbf9e5a0aa7a9 P 3ef63aea111c483faa410fa692f3eff2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:09:02.166651 17588 catalog_manager.cc:5484] T 33d83caedf904c7ba1bdbf9e5a0aa7a9 P 3ef63aea111c483faa410fa692f3eff2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ef63aea111c483faa410fa692f3eff2 (127.16.13.195). New cstate: current_term: 1 leader_uuid: "3ef63aea111c483faa410fa692f3eff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ef63aea111c483faa410fa692f3eff2" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 38489 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:02.168521 18011 raft_consensus.cc:348] T a5245af584e84fd295008a34a804da1c P 31663ec98b49478b84f80767cfe3f33e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31663ec98b49478b84f80767cfe3f33e" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 44673 } }
I20240501 14:09:02.168632 18011 raft_consensus.cc:374] T a5245af584e84fd295008a34a804da1c P 31663ec98b49478b84f80767cfe3f33e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:02.168671 18011 raft_consensus.cc:729] T a5245af584e84fd295008a34a804da1c P 31663ec98b49478b84f80767cfe3f33e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31663ec98b49478b84f80767cfe3f33e, State: Initialized, Role: FOLLOWER
I20240501 14:09:02.168823 18011 consensus_queue.cc:260] T a5245af584e84fd295008a34a804da1c P 31663ec98b49478b84f80767cfe3f33e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31663ec98b49478b84f80767cfe3f33e" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 44673 } }
I20240501 14:09:02.168910 18011 raft_consensus.cc:388] T a5245af584e84fd295008a34a804da1c P 31663ec98b49478b84f80767cfe3f33e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:09:02.168951 18011 raft_consensus.cc:482] T a5245af584e84fd295008a34a804da1c P 31663ec98b49478b84f80767cfe3f33e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:09:02.169001 18011 raft_consensus.cc:3037] T a5245af584e84fd295008a34a804da1c P 31663ec98b49478b84f80767cfe3f33e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:02.170542 18011 raft_consensus.cc:504] T a5245af584e84fd295008a34a804da1c P 31663ec98b49478b84f80767cfe3f33e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31663ec98b49478b84f80767cfe3f33e" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 44673 } }
I20240501 14:09:02.170689 18011 leader_election.cc:302] T a5245af584e84fd295008a34a804da1c P 31663ec98b49478b84f80767cfe3f33e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31663ec98b49478b84f80767cfe3f33e; no voters:
I20240501 14:09:02.170910 18011 leader_election.cc:288] T a5245af584e84fd295008a34a804da1c P 31663ec98b49478b84f80767cfe3f33e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:09:02.171252 18011 ts_tablet_manager.cc:1398] T a5245af584e84fd295008a34a804da1c P 31663ec98b49478b84f80767cfe3f33e: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20240501 14:09:02.173278 18014 tablet_bootstrap.cc:492] T 42fe800961734151bb412c007d1d5b99 P 9b95286c84c84e99aefecb5d5a5fb345: No bootstrap required, opened a new log
I20240501 14:09:02.173540 18014 ts_tablet_manager.cc:1367] T 42fe800961734151bb412c007d1d5b99 P 9b95286c84c84e99aefecb5d5a5fb345: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 14:09:02.174160 18014 raft_consensus.cc:348] T 42fe800961734151bb412c007d1d5b99 P 9b95286c84c84e99aefecb5d5a5fb345 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b95286c84c84e99aefecb5d5a5fb345" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33927 } }
I20240501 14:09:02.174326 18014 raft_consensus.cc:374] T 42fe800961734151bb412c007d1d5b99 P 9b95286c84c84e99aefecb5d5a5fb345 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:02.174417 18014 raft_consensus.cc:729] T 42fe800961734151bb412c007d1d5b99 P 9b95286c84c84e99aefecb5d5a5fb345 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b95286c84c84e99aefecb5d5a5fb345, State: Initialized, Role: FOLLOWER
I20240501 14:09:02.174638 18014 consensus_queue.cc:260] T 42fe800961734151bb412c007d1d5b99 P 9b95286c84c84e99aefecb5d5a5fb345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b95286c84c84e99aefecb5d5a5fb345" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33927 } }
I20240501 14:09:02.174793 18014 raft_consensus.cc:388] T 42fe800961734151bb412c007d1d5b99 P 9b95286c84c84e99aefecb5d5a5fb345 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:09:02.174880 18014 raft_consensus.cc:482] T 42fe800961734151bb412c007d1d5b99 P 9b95286c84c84e99aefecb5d5a5fb345 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:09:02.174978 18014 raft_consensus.cc:3037] T 42fe800961734151bb412c007d1d5b99 P 9b95286c84c84e99aefecb5d5a5fb345 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:02.176048 18021 raft_consensus.cc:2781] T a5245af584e84fd295008a34a804da1c P 31663ec98b49478b84f80767cfe3f33e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:02.176183 18014 raft_consensus.cc:504] T 42fe800961734151bb412c007d1d5b99 P 9b95286c84c84e99aefecb5d5a5fb345 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b95286c84c84e99aefecb5d5a5fb345" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33927 } }
I20240501 14:09:02.176307 18021 raft_consensus.cc:686] T a5245af584e84fd295008a34a804da1c P 31663ec98b49478b84f80767cfe3f33e [term 1 LEADER]: Becoming Leader. State: Replica: 31663ec98b49478b84f80767cfe3f33e, State: Running, Role: LEADER
I20240501 14:09:02.176313 18014 leader_election.cc:302] T 42fe800961734151bb412c007d1d5b99 P 9b95286c84c84e99aefecb5d5a5fb345 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b95286c84c84e99aefecb5d5a5fb345; no voters:
I20240501 14:09:02.176436 18021 consensus_queue.cc:237] T a5245af584e84fd295008a34a804da1c P 31663ec98b49478b84f80767cfe3f33e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31663ec98b49478b84f80767cfe3f33e" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 44673 } }
I20240501 14:09:02.176739 18014 leader_election.cc:288] T 42fe800961734151bb412c007d1d5b99 P 9b95286c84c84e99aefecb5d5a5fb345 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:09:02.176821 18024 raft_consensus.cc:2781] T 42fe800961734151bb412c007d1d5b99 P 9b95286c84c84e99aefecb5d5a5fb345 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:02.177006 18024 raft_consensus.cc:686] T 42fe800961734151bb412c007d1d5b99 P 9b95286c84c84e99aefecb5d5a5fb345 [term 1 LEADER]: Becoming Leader. State: Replica: 9b95286c84c84e99aefecb5d5a5fb345, State: Running, Role: LEADER
I20240501 14:09:02.177096 18014 ts_tablet_manager.cc:1398] T 42fe800961734151bb412c007d1d5b99 P 9b95286c84c84e99aefecb5d5a5fb345: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:09:02.177124 18024 consensus_queue.cc:237] T 42fe800961734151bb412c007d1d5b99 P 9b95286c84c84e99aefecb5d5a5fb345 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b95286c84c84e99aefecb5d5a5fb345" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33927 } }
I20240501 14:09:02.178011 17588 catalog_manager.cc:5484] T a5245af584e84fd295008a34a804da1c P 31663ec98b49478b84f80767cfe3f33e reported cstate change: term changed from 0 to 1, leader changed from <none> to 31663ec98b49478b84f80767cfe3f33e (127.16.13.194). New cstate: current_term: 1 leader_uuid: "31663ec98b49478b84f80767cfe3f33e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31663ec98b49478b84f80767cfe3f33e" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 44673 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:02.179147 17587 catalog_manager.cc:5484] T 42fe800961734151bb412c007d1d5b99 P 9b95286c84c84e99aefecb5d5a5fb345 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b95286c84c84e99aefecb5d5a5fb345 (127.16.13.193). New cstate: current_term: 1 leader_uuid: "9b95286c84c84e99aefecb5d5a5fb345" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b95286c84c84e99aefecb5d5a5fb345" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 33927 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:02.190572 16439 master.cc:537] Master@127.16.13.253:39083 shutting down...
I20240501 14:09:02.194396 16439 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:02.194665 16439 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:02.194749 16439 tablet_replica.cc:330] T 00000000000000000000000000000000 P 33806aa6a42e403e84358aa7fe3e63c7: stopping tablet replica
I20240501 14:09:02.218693 16439 master.cc:552] Master@127.16.13.253:39083 shutdown complete.
W20240501 14:09:03.177255 17977 heartbeater.cc:644] Failed to heartbeat to 127.16.13.253:39083 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.13.253:39083: connect: Connection refused (error 111)
I20240501 14:09:03.685075 18037 raft_consensus.cc:482] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 33806aa6a42e403e84358aa7fe3e63c7)
I20240501 14:09:03.685243 18037 raft_consensus.cc:504] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76fcf8e69df45e39d4c4f51fb73be56" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 33301 } } peers { permanent_uuid: "33806aa6a42e403e84358aa7fe3e63c7" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 39083 } } peers { permanent_uuid: "b34a588c4a864adca592f5081c252e99" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 42471 } }
I20240501 14:09:03.685703 18037 leader_election.cc:288] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 33806aa6a42e403e84358aa7fe3e63c7 (127.16.13.253:39083), b34a588c4a864adca592f5081c252e99 (127.16.13.252:42471)
I20240501 14:09:03.686142 17664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f76fcf8e69df45e39d4c4f51fb73be56" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "b34a588c4a864adca592f5081c252e99" is_pre_election: true
I20240501 14:09:03.686275 17664 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b34a588c4a864adca592f5081c252e99 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f76fcf8e69df45e39d4c4f51fb73be56 in term 1.
I20240501 14:09:03.686599 17509 leader_election.cc:302] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b34a588c4a864adca592f5081c252e99, f76fcf8e69df45e39d4c4f51fb73be56; no voters:
I20240501 14:09:03.686800 18037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:09:03.686848 18037 raft_consensus.cc:482] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 33806aa6a42e403e84358aa7fe3e63c7)
W20240501 14:09:03.686904 17506 leader_election.cc:334] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33806aa6a42e403e84358aa7fe3e63c7 (127.16.13.253:39083): Network error: Client connection negotiation failed: client connection to 127.16.13.253:39083: connect: Connection refused (error 111)
I20240501 14:09:03.686939 18037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:03.687819 18037 raft_consensus.cc:504] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76fcf8e69df45e39d4c4f51fb73be56" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 33301 } } peers { permanent_uuid: "33806aa6a42e403e84358aa7fe3e63c7" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 39083 } } peers { permanent_uuid: "b34a588c4a864adca592f5081c252e99" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 42471 } }
I20240501 14:09:03.688212 18037 leader_election.cc:288] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [CANDIDATE]: Term 2 election: Requested vote from peers 33806aa6a42e403e84358aa7fe3e63c7 (127.16.13.253:39083), b34a588c4a864adca592f5081c252e99 (127.16.13.252:42471)
I20240501 14:09:03.688534 17664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f76fcf8e69df45e39d4c4f51fb73be56" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "b34a588c4a864adca592f5081c252e99"
I20240501 14:09:03.688637 17664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b34a588c4a864adca592f5081c252e99 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:09:03.689000 17506 leader_election.cc:334] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 33806aa6a42e403e84358aa7fe3e63c7 (127.16.13.253:39083): Network error: Client connection negotiation failed: client connection to 127.16.13.253:39083: connect: Connection refused (error 111)
I20240501 14:09:03.689675 17664 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b34a588c4a864adca592f5081c252e99 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f76fcf8e69df45e39d4c4f51fb73be56 in term 2.
I20240501 14:09:03.689939 17509 leader_election.cc:302] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b34a588c4a864adca592f5081c252e99, f76fcf8e69df45e39d4c4f51fb73be56; no voters: 33806aa6a42e403e84358aa7fe3e63c7
I20240501 14:09:03.690083 18037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:03.690259 18037 raft_consensus.cc:686] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [term 2 LEADER]: Becoming Leader. State: Replica: f76fcf8e69df45e39d4c4f51fb73be56, State: Running, Role: LEADER
I20240501 14:09:03.690393 18037 consensus_queue.cc:237] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76fcf8e69df45e39d4c4f51fb73be56" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 33301 } } peers { permanent_uuid: "33806aa6a42e403e84358aa7fe3e63c7" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 39083 } } peers { permanent_uuid: "b34a588c4a864adca592f5081c252e99" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 42471 } }
I20240501 14:09:03.690979 18040 sys_catalog.cc:455] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f76fcf8e69df45e39d4c4f51fb73be56. Latest consensus state: current_term: 2 leader_uuid: "f76fcf8e69df45e39d4c4f51fb73be56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76fcf8e69df45e39d4c4f51fb73be56" member_type: VOTER last_known_addr { host: "127.16.13.254" port: 33301 } } peers { permanent_uuid: "33806aa6a42e403e84358aa7fe3e63c7" member_type: VOTER last_known_addr { host: "127.16.13.253" port: 39083 } } peers { permanent_uuid: "b34a588c4a864adca592f5081c252e99" member_type: VOTER last_known_addr { host: "127.16.13.252" port: 42471 } } }
I20240501 14:09:03.691066 18040 sys_catalog.cc:458] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [sys.catalog]: This master's current role is: LEADER
I20240501 14:09:03.691280 18042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:09:03.692355 18042 catalog_manager.cc:669] Loaded metadata for table test-workload [id=ff9b6cac3ae949268f57f34724632176]
I20240501 14:09:03.693012 18042 tablet_loader.cc:96] loaded metadata for tablet 33d83caedf904c7ba1bdbf9e5a0aa7a9 (table test-workload [id=ff9b6cac3ae949268f57f34724632176])
I20240501 14:09:03.693183 18042 tablet_loader.cc:96] loaded metadata for tablet 42fe800961734151bb412c007d1d5b99 (table test-workload [id=ff9b6cac3ae949268f57f34724632176])
I20240501 14:09:03.693312 18042 tablet_loader.cc:96] loaded metadata for tablet a5245af584e84fd295008a34a804da1c (table test-workload [id=ff9b6cac3ae949268f57f34724632176])
I20240501 14:09:03.693380 18042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:09:03.693689 18042 catalog_manager.cc:1259] Loaded cluster ID: 432f4f7750ab41ea97beb1f10fe613b5
I20240501 14:09:03.693728 18042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:09:03.694231 18042 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:09:03.694582 18042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56: Loaded TSK: 0
I20240501 14:09:03.694972 18042 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:09:03.953436 17731 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:09:03.953560 17731 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:09:03.984769 16439 tablet_server.cc:178] TabletServer@127.16.13.193:0 shutting down...
I20240501 14:09:03.988775 16439 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:09:03.988973 16439 tablet_replica.cc:330] T 42fe800961734151bb412c007d1d5b99 P 9b95286c84c84e99aefecb5d5a5fb345: stopping tablet replica
I20240501 14:09:03.989089 16439 raft_consensus.cc:2229] T 42fe800961734151bb412c007d1d5b99 P 9b95286c84c84e99aefecb5d5a5fb345 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:03.989216 16439 raft_consensus.cc:2258] T 42fe800961734151bb412c007d1d5b99 P 9b95286c84c84e99aefecb5d5a5fb345 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:04.001760 16439 tablet_server.cc:194] TabletServer@127.16.13.193:0 shutdown complete.
I20240501 14:09:04.004053 16439 tablet_server.cc:178] TabletServer@127.16.13.194:0 shutting down...
I20240501 14:09:04.008359 16439 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:09:04.008543 16439 tablet_replica.cc:330] T a5245af584e84fd295008a34a804da1c P 31663ec98b49478b84f80767cfe3f33e: stopping tablet replica
I20240501 14:09:04.008656 16439 raft_consensus.cc:2229] T a5245af584e84fd295008a34a804da1c P 31663ec98b49478b84f80767cfe3f33e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:04.008757 16439 raft_consensus.cc:2258] T a5245af584e84fd295008a34a804da1c P 31663ec98b49478b84f80767cfe3f33e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:04.021026 16439 tablet_server.cc:194] TabletServer@127.16.13.194:0 shutdown complete.
I20240501 14:09:04.022907 16439 tablet_server.cc:178] TabletServer@127.16.13.195:0 shutting down...
I20240501 14:09:04.027345 16439 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:09:04.027534 16439 tablet_replica.cc:330] T 33d83caedf904c7ba1bdbf9e5a0aa7a9 P 3ef63aea111c483faa410fa692f3eff2: stopping tablet replica
I20240501 14:09:04.027653 16439 raft_consensus.cc:2229] T 33d83caedf904c7ba1bdbf9e5a0aa7a9 P 3ef63aea111c483faa410fa692f3eff2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:04.027843 16439 raft_consensus.cc:2258] T 33d83caedf904c7ba1bdbf9e5a0aa7a9 P 3ef63aea111c483faa410fa692f3eff2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:04.040288 16439 tablet_server.cc:194] TabletServer@127.16.13.195:0 shutdown complete.
I20240501 14:09:04.042207 16439 master.cc:537] Master@127.16.13.254:33301 shutting down...
I20240501 14:09:04.045890 16439 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [term 2 LEADER]: Raft consensus shutting down.
I20240501 14:09:04.046082 16439 pending_rounds.cc:62] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56: Trying to abort 1 pending ops.
I20240501 14:09:04.046139 16439 pending_rounds.cc:68] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56: Aborting op as it isn't in flight: id { term: 2 index: 10 } timestamp: 7022889138957225984 op_type: NO_OP noop_request { }
I20240501 14:09:04.046226 16439 raft_consensus.cc:2866] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240501 14:09:04.046264 16439 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:04.046339 16439 tablet_replica.cc:330] T 00000000000000000000000000000000 P f76fcf8e69df45e39d4c4f51fb73be56: stopping tablet replica
I20240501 14:09:04.060148 16439 master.cc:552] Master@127.16.13.254:33301 shutdown complete.
I20240501 14:09:04.063058 16439 master.cc:537] Master@127.16.13.252:42471 shutting down...
I20240501 14:09:04.066763 16439 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b34a588c4a864adca592f5081c252e99 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:04.066920 16439 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b34a588c4a864adca592f5081c252e99 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:04.066987 16439 tablet_replica.cc:330] T 00000000000000000000000000000000 P b34a588c4a864adca592f5081c252e99: stopping tablet replica
I20240501 14:09:04.079476 16439 master.cc:552] Master@127.16.13.252:42471 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (4279 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20240501 14:09:04.086767 16439 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.13.254:42829
I20240501 14:09:04.087098 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:04.089759 18044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:04.090512 18046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:04.091576 18043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:04.092270 16439 server_base.cc:1034] running on GCE node
I20240501 14:09:04.092437 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:09:04.092475 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:09:04.092504 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572544092503 us; error 0 us; skew 500 ppm
I20240501 14:09:04.093488 16439 webserver.cc:460] Webserver started at http://127.16.13.254:44885/ using document root <none> and password file <none>
I20240501 14:09:04.093638 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:04.093703 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:04.093791 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:04.094215 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714572529860089-16439-0/minicluster-data/master-0-root/instance:
uuid: "77b8eff2377b453b94d0789d717c0a05"
format_stamp: "Formatted at 2024-05-01 14:09:04 on dist-test-slave-mgxc"
I20240501 14:09:04.095711 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240501 14:09:04.096796 18051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:04.097087 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:04.097158 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714572529860089-16439-0/minicluster-data/master-0-root
uuid: "77b8eff2377b453b94d0789d717c0a05"
format_stamp: "Formatted at 2024-05-01 14:09:04 on dist-test-slave-mgxc"
I20240501 14:09:04.097234 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714572529860089-16439-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714572529860089-16439-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714572529860089-16439-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:04.101542 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:04.101848 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:04.105754 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.254:42829
I20240501 14:09:04.109009 18103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.254:42829 every 8 connection(s)
I20240501 14:09:04.109442 18104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:04.111248 18104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77b8eff2377b453b94d0789d717c0a05: Bootstrap starting.
I20240501 14:09:04.112071 18104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77b8eff2377b453b94d0789d717c0a05: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:04.113137 18104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77b8eff2377b453b94d0789d717c0a05: No bootstrap required, opened a new log
I20240501 14:09:04.113590 18104 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77b8eff2377b453b94d0789d717c0a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77b8eff2377b453b94d0789d717c0a05" member_type: VOTER }
I20240501 14:09:04.113700 18104 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77b8eff2377b453b94d0789d717c0a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:04.113736 18104 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77b8eff2377b453b94d0789d717c0a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b8eff2377b453b94d0789d717c0a05, State: Initialized, Role: FOLLOWER
I20240501 14:09:04.113895 18104 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77b8eff2377b453b94d0789d717c0a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77b8eff2377b453b94d0789d717c0a05" member_type: VOTER }
I20240501 14:09:04.114002 18104 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77b8eff2377b453b94d0789d717c0a05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:09:04.114054 18104 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77b8eff2377b453b94d0789d717c0a05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:09:04.114109 18104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77b8eff2377b453b94d0789d717c0a05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:04.114899 18104 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77b8eff2377b453b94d0789d717c0a05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77b8eff2377b453b94d0789d717c0a05" member_type: VOTER }
I20240501 14:09:04.115011 18104 leader_election.cc:302] T 00000000000000000000000000000000 P 77b8eff2377b453b94d0789d717c0a05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77b8eff2377b453b94d0789d717c0a05; no voters:
I20240501 14:09:04.115165 18104 leader_election.cc:288] T 00000000000000000000000000000000 P 77b8eff2377b453b94d0789d717c0a05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:09:04.115301 18107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77b8eff2377b453b94d0789d717c0a05 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:04.115494 18107 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77b8eff2377b453b94d0789d717c0a05 [term 1 LEADER]: Becoming Leader. State: Replica: 77b8eff2377b453b94d0789d717c0a05, State: Running, Role: LEADER
I20240501 14:09:04.115592 18104 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77b8eff2377b453b94d0789d717c0a05 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:09:04.115612 18107 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77b8eff2377b453b94d0789d717c0a05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77b8eff2377b453b94d0789d717c0a05" member_type: VOTER }
I20240501 14:09:04.116036 18108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77b8eff2377b453b94d0789d717c0a05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77b8eff2377b453b94d0789d717c0a05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77b8eff2377b453b94d0789d717c0a05" member_type: VOTER } }
I20240501 14:09:04.116052 18109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77b8eff2377b453b94d0789d717c0a05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77b8eff2377b453b94d0789d717c0a05. Latest consensus state: current_term: 1 leader_uuid: "77b8eff2377b453b94d0789d717c0a05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77b8eff2377b453b94d0789d717c0a05" member_type: VOTER } }
I20240501 14:09:04.116178 18109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77b8eff2377b453b94d0789d717c0a05 [sys.catalog]: This master's current role is: LEADER
I20240501 14:09:04.116366 18108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77b8eff2377b453b94d0789d717c0a05 [sys.catalog]: This master's current role is: LEADER
I20240501 14:09:04.116782 18112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:09:04.117523 18112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:09:04.117681 16439 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 14:09:04.119357 18112 catalog_manager.cc:1347] Generated new cluster ID: aa0b8f9d68f94450b8d5e891cb7ec8f2
I20240501 14:09:04.119412 18112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:09:04.151665 18112 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:09:04.152295 18112 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:09:04.183231 18112 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 77b8eff2377b453b94d0789d717c0a05: Generated new TSK 0
I20240501 14:09:04.183422 18112 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:09:04.246649 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:04.249682 18125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:04.250370 18128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:04.251370 18126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:04.252120 16439 server_base.cc:1034] running on GCE node
I20240501 14:09:04.252334 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:09:04.252380 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:09:04.252403 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572544252402 us; error 0 us; skew 500 ppm
I20240501 14:09:04.253466 16439 webserver.cc:460] Webserver started at http://127.16.13.193:38153/ using document root <none> and password file <none>
I20240501 14:09:04.253610 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:04.253664 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:04.253752 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:04.254186 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714572529860089-16439-0/minicluster-data/ts-0-root/instance:
uuid: "1d83861e0ee04f30a7c78a55a2ddd33e"
format_stamp: "Formatted at 2024-05-01 14:09:04 on dist-test-slave-mgxc"
I20240501 14:09:04.255726 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240501 14:09:04.256722 18133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:04.256973 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:04.257032 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714572529860089-16439-0/minicluster-data/ts-0-root
uuid: "1d83861e0ee04f30a7c78a55a2ddd33e"
format_stamp: "Formatted at 2024-05-01 14:09:04 on dist-test-slave-mgxc"
I20240501 14:09:04.257089 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714572529860089-16439-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714572529860089-16439-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714572529860089-16439-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:04.269942 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:04.270328 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:04.270568 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:04.271010 16439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:04.271049 16439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:04.271080 16439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:04.271109 16439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:04.275450 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.193:42739
I20240501 14:09:04.275499 18196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.193:42739 every 8 connection(s)
I20240501 14:09:04.276284 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:04.279145 18204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:04.280103 18202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:04.280606 18197 heartbeater.cc:347] Connected to a master server at 127.16.13.254:42829
I20240501 14:09:04.280702 18197 heartbeater.cc:464] Registering TS with master...
W20240501 14:09:04.280797 18201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:04.280939 18197 heartbeater.cc:511] Master 127.16.13.254:42829 requested a full tablet report, sending...
I20240501 14:09:04.281687 18068 ts_manager.cc:194] Registered new tserver with Master: 1d83861e0ee04f30a7c78a55a2ddd33e (127.16.13.193:42739)
I20240501 14:09:04.281781 16439 server_base.cc:1034] running on GCE node
I20240501 14:09:04.282052 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:09:04.282094 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:09:04.282128 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572544282127 us; error 0 us; skew 500 ppm
I20240501 14:09:04.282673 18068 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59482
I20240501 14:09:04.283319 16439 webserver.cc:460] Webserver started at http://127.16.13.194:44647/ using document root <none> and password file <none>
I20240501 14:09:04.283524 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:04.283586 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:04.283644 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:04.284032 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714572529860089-16439-0/minicluster-data/ts-1-root/instance:
uuid: "432945db49b84e008b12c241c76325ed"
format_stamp: "Formatted at 2024-05-01 14:09:04 on dist-test-slave-mgxc"
I20240501 14:09:04.285465 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:04.286422 18209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:04.286674 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:04.286737 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714572529860089-16439-0/minicluster-data/ts-1-root
uuid: "432945db49b84e008b12c241c76325ed"
format_stamp: "Formatted at 2024-05-01 14:09:04 on dist-test-slave-mgxc"
I20240501 14:09:04.286796 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714572529860089-16439-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714572529860089-16439-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714572529860089-16439-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:04.311678 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:04.312103 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:04.312379 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:04.312902 16439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:04.312942 16439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:04.312979 16439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:04.313019 16439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:04.317150 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.194:35635
I20240501 14:09:04.317229 18272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.194:35635 every 8 connection(s)
I20240501 14:09:04.317927 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:04.320820 18277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:04.321477 18276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:04.322376 18273 heartbeater.cc:347] Connected to a master server at 127.16.13.254:42829
I20240501 14:09:04.322408 16439 server_base.cc:1034] running on GCE node
I20240501 14:09:04.322499 18273 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:04.322687 18273 heartbeater.cc:511] Master 127.16.13.254:42829 requested a full tablet report, sending...
W20240501 14:09:04.323187 18279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:04.323323 18068 ts_manager.cc:194] Registered new tserver with Master: 432945db49b84e008b12c241c76325ed (127.16.13.194:35635)
I20240501 14:09:04.323419 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:09:04.323465 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:09:04.323495 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572544323495 us; error 0 us; skew 500 ppm
I20240501 14:09:04.324030 18068 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59484
I20240501 14:09:04.324555 16439 webserver.cc:460] Webserver started at http://127.16.13.195:41939/ using document root <none> and password file <none>
I20240501 14:09:04.324720 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:04.324784 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:04.324857 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:04.325270 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714572529860089-16439-0/minicluster-data/ts-2-root/instance:
uuid: "eebf4bf4a2224758a12fc26f6a87aa95"
format_stamp: "Formatted at 2024-05-01 14:09:04 on dist-test-slave-mgxc"
I20240501 14:09:04.326686 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:04.327605 18284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:04.327852 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:04.327952 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714572529860089-16439-0/minicluster-data/ts-2-root
uuid: "eebf4bf4a2224758a12fc26f6a87aa95"
format_stamp: "Formatted at 2024-05-01 14:09:04 on dist-test-slave-mgxc"
I20240501 14:09:04.328020 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714572529860089-16439-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714572529860089-16439-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714572529860089-16439-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:04.340528 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:04.340934 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:04.341239 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:04.341688 16439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:04.341727 16439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:04.341770 16439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:04.341800 16439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:04.346166 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.195:43181
I20240501 14:09:04.346341 18347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.195:43181 every 8 connection(s)
I20240501 14:09:04.350621 18348 heartbeater.cc:347] Connected to a master server at 127.16.13.254:42829
I20240501 14:09:04.350718 18348 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:04.350916 18348 heartbeater.cc:511] Master 127.16.13.254:42829 requested a full tablet report, sending...
I20240501 14:09:04.351552 18068 ts_manager.cc:194] Registered new tserver with Master: eebf4bf4a2224758a12fc26f6a87aa95 (127.16.13.195:43181)
I20240501 14:09:04.352137 18068 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59486
I20240501 14:09:04.352188 16439 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005602483s
I20240501 14:09:05.283705 18197 heartbeater.cc:503] Master 127.16.13.254:42829 was elected leader, sending a full tablet report...
I20240501 14:09:05.325062 18273 heartbeater.cc:503] Master 127.16.13.254:42829 was elected leader, sending a full tablet report...
I20240501 14:09:05.353099 18348 heartbeater.cc:503] Master 127.16.13.254:42829 was elected leader, sending a full tablet report...
I20240501 14:09:05.378314 16439 test_util.cc:268] Using random seed: 125978630
I20240501 14:09:05.389586 18068 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59500:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 14:09:05.390010 18068 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 14:09:05.399539 18312 tablet_service.cc:1467] Processing CreateTablet for tablet 001cf18440d945b6824493c39b1d7487 (DEFAULT_TABLE table=test-workload [id=9df3269fbabd4e04aa4aa4e32b8797fc]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240501 14:09:05.399717 18237 tablet_service.cc:1467] Processing CreateTablet for tablet 001cf18440d945b6824493c39b1d7487 (DEFAULT_TABLE table=test-workload [id=9df3269fbabd4e04aa4aa4e32b8797fc]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240501 14:09:05.399870 18312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 001cf18440d945b6824493c39b1d7487. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:05.400014 18237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 001cf18440d945b6824493c39b1d7487. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:05.402010 18368 tablet_bootstrap.cc:492] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95: Bootstrap starting.
I20240501 14:09:05.402052 18369 tablet_bootstrap.cc:492] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed: Bootstrap starting.
I20240501 14:09:05.402176 18311 tablet_service.cc:1467] Processing CreateTablet for tablet 78bc2976729d4670aab390265c3aca4a (DEFAULT_TABLE table=test-workload [id=9df3269fbabd4e04aa4aa4e32b8797fc]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240501 14:09:05.402233 18236 tablet_service.cc:1467] Processing CreateTablet for tablet 78bc2976729d4670aab390265c3aca4a (DEFAULT_TABLE table=test-workload [id=9df3269fbabd4e04aa4aa4e32b8797fc]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240501 14:09:05.402472 18311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78bc2976729d4670aab390265c3aca4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:05.402863 18161 tablet_service.cc:1467] Processing CreateTablet for tablet 001cf18440d945b6824493c39b1d7487 (DEFAULT_TABLE table=test-workload [id=9df3269fbabd4e04aa4aa4e32b8797fc]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240501 14:09:05.402992 18159 tablet_service.cc:1467] Processing CreateTablet for tablet 78bc2976729d4670aab390265c3aca4a (DEFAULT_TABLE table=test-workload [id=9df3269fbabd4e04aa4aa4e32b8797fc]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240501 14:09:05.403092 18161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 001cf18440d945b6824493c39b1d7487. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:05.403167 18368 tablet_bootstrap.cc:654] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:05.403249 18159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78bc2976729d4670aab390265c3aca4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:05.405299 18371 tablet_bootstrap.cc:492] T 78bc2976729d4670aab390265c3aca4a P 1d83861e0ee04f30a7c78a55a2ddd33e: Bootstrap starting.
I20240501 14:09:05.402479 18236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78bc2976729d4670aab390265c3aca4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:05.405895 18368 tablet_bootstrap.cc:492] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95: No bootstrap required, opened a new log
I20240501 14:09:05.405988 18368 ts_tablet_manager.cc:1367] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:05.406261 18371 tablet_bootstrap.cc:654] T 78bc2976729d4670aab390265c3aca4a P 1d83861e0ee04f30a7c78a55a2ddd33e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:05.406428 18368 raft_consensus.cc:348] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } }
I20240501 14:09:05.406538 18368 raft_consensus.cc:374] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:05.406575 18368 raft_consensus.cc:729] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebf4bf4a2224758a12fc26f6a87aa95, State: Initialized, Role: FOLLOWER
I20240501 14:09:05.406742 18368 consensus_queue.cc:260] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } }
I20240501 14:09:05.406998 18369 tablet_bootstrap.cc:654] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:05.407564 18368 ts_tablet_manager.cc:1398] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:05.407749 18368 tablet_bootstrap.cc:492] T 78bc2976729d4670aab390265c3aca4a P eebf4bf4a2224758a12fc26f6a87aa95: Bootstrap starting.
I20240501 14:09:05.408176 18369 tablet_bootstrap.cc:492] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed: No bootstrap required, opened a new log
I20240501 14:09:05.408340 18369 ts_tablet_manager.cc:1367] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:05.408674 18368 tablet_bootstrap.cc:654] T 78bc2976729d4670aab390265c3aca4a P eebf4bf4a2224758a12fc26f6a87aa95: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:05.408912 18369 raft_consensus.cc:348] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } }
I20240501 14:09:05.409081 18369 raft_consensus.cc:374] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:05.409169 18369 raft_consensus.cc:729] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 432945db49b84e008b12c241c76325ed, State: Initialized, Role: FOLLOWER
I20240501 14:09:05.408339 18371 tablet_bootstrap.cc:492] T 78bc2976729d4670aab390265c3aca4a P 1d83861e0ee04f30a7c78a55a2ddd33e: No bootstrap required, opened a new log
I20240501 14:09:05.409354 18371 ts_tablet_manager.cc:1367] T 78bc2976729d4670aab390265c3aca4a P 1d83861e0ee04f30a7c78a55a2ddd33e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:05.409442 18369 consensus_queue.cc:260] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } }
I20240501 14:09:05.409812 18369 ts_tablet_manager.cc:1398] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:05.409795 18371 raft_consensus.cc:348] T 78bc2976729d4670aab390265c3aca4a P 1d83861e0ee04f30a7c78a55a2ddd33e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } } peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } }
I20240501 14:09:05.409904 18371 raft_consensus.cc:374] T 78bc2976729d4670aab390265c3aca4a P 1d83861e0ee04f30a7c78a55a2ddd33e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:05.409943 18371 raft_consensus.cc:729] T 78bc2976729d4670aab390265c3aca4a P 1d83861e0ee04f30a7c78a55a2ddd33e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d83861e0ee04f30a7c78a55a2ddd33e, State: Initialized, Role: FOLLOWER
I20240501 14:09:05.410013 18369 tablet_bootstrap.cc:492] T 78bc2976729d4670aab390265c3aca4a P 432945db49b84e008b12c241c76325ed: Bootstrap starting.
I20240501 14:09:05.410071 18371 consensus_queue.cc:260] T 78bc2976729d4670aab390265c3aca4a P 1d83861e0ee04f30a7c78a55a2ddd33e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } } peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } }
I20240501 14:09:05.410405 18371 ts_tablet_manager.cc:1398] T 78bc2976729d4670aab390265c3aca4a P 1d83861e0ee04f30a7c78a55a2ddd33e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:05.410571 18371 tablet_bootstrap.cc:492] T 001cf18440d945b6824493c39b1d7487 P 1d83861e0ee04f30a7c78a55a2ddd33e: Bootstrap starting.
I20240501 14:09:05.411203 18369 tablet_bootstrap.cc:654] T 78bc2976729d4670aab390265c3aca4a P 432945db49b84e008b12c241c76325ed: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:05.411448 18371 tablet_bootstrap.cc:654] T 001cf18440d945b6824493c39b1d7487 P 1d83861e0ee04f30a7c78a55a2ddd33e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:05.412175 18368 tablet_bootstrap.cc:492] T 78bc2976729d4670aab390265c3aca4a P eebf4bf4a2224758a12fc26f6a87aa95: No bootstrap required, opened a new log
I20240501 14:09:05.412257 18368 ts_tablet_manager.cc:1367] T 78bc2976729d4670aab390265c3aca4a P eebf4bf4a2224758a12fc26f6a87aa95: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:05.412657 18368 raft_consensus.cc:348] T 78bc2976729d4670aab390265c3aca4a P eebf4bf4a2224758a12fc26f6a87aa95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } } peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } }
I20240501 14:09:05.412751 18371 tablet_bootstrap.cc:492] T 001cf18440d945b6824493c39b1d7487 P 1d83861e0ee04f30a7c78a55a2ddd33e: No bootstrap required, opened a new log
I20240501 14:09:05.412762 18368 raft_consensus.cc:374] T 78bc2976729d4670aab390265c3aca4a P eebf4bf4a2224758a12fc26f6a87aa95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:05.412850 18371 ts_tablet_manager.cc:1367] T 001cf18440d945b6824493c39b1d7487 P 1d83861e0ee04f30a7c78a55a2ddd33e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:05.413412 18371 raft_consensus.cc:348] T 001cf18440d945b6824493c39b1d7487 P 1d83861e0ee04f30a7c78a55a2ddd33e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } }
I20240501 14:09:05.413512 18371 raft_consensus.cc:374] T 001cf18440d945b6824493c39b1d7487 P 1d83861e0ee04f30a7c78a55a2ddd33e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:05.413544 18371 raft_consensus.cc:729] T 001cf18440d945b6824493c39b1d7487 P 1d83861e0ee04f30a7c78a55a2ddd33e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d83861e0ee04f30a7c78a55a2ddd33e, State: Initialized, Role: FOLLOWER
I20240501 14:09:05.413650 18371 consensus_queue.cc:260] T 001cf18440d945b6824493c39b1d7487 P 1d83861e0ee04f30a7c78a55a2ddd33e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } }
I20240501 14:09:05.413718 18369 tablet_bootstrap.cc:492] T 78bc2976729d4670aab390265c3aca4a P 432945db49b84e008b12c241c76325ed: No bootstrap required, opened a new log
I20240501 14:09:05.413784 18369 ts_tablet_manager.cc:1367] T 78bc2976729d4670aab390265c3aca4a P 432945db49b84e008b12c241c76325ed: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:05.413910 18371 ts_tablet_manager.cc:1398] T 001cf18440d945b6824493c39b1d7487 P 1d83861e0ee04f30a7c78a55a2ddd33e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:05.412838 18368 raft_consensus.cc:729] T 78bc2976729d4670aab390265c3aca4a P eebf4bf4a2224758a12fc26f6a87aa95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebf4bf4a2224758a12fc26f6a87aa95, State: Initialized, Role: FOLLOWER
I20240501 14:09:05.414188 18369 raft_consensus.cc:348] T 78bc2976729d4670aab390265c3aca4a P 432945db49b84e008b12c241c76325ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } } peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } }
I20240501 14:09:05.414563 18369 raft_consensus.cc:374] T 78bc2976729d4670aab390265c3aca4a P 432945db49b84e008b12c241c76325ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:05.414597 18369 raft_consensus.cc:729] T 78bc2976729d4670aab390265c3aca4a P 432945db49b84e008b12c241c76325ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 432945db49b84e008b12c241c76325ed, State: Initialized, Role: FOLLOWER
I20240501 14:09:05.414712 18369 consensus_queue.cc:260] T 78bc2976729d4670aab390265c3aca4a P 432945db49b84e008b12c241c76325ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } } peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } }
I20240501 14:09:05.414969 18369 ts_tablet_manager.cc:1398] T 78bc2976729d4670aab390265c3aca4a P 432945db49b84e008b12c241c76325ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:05.414265 18368 consensus_queue.cc:260] T 78bc2976729d4670aab390265c3aca4a P eebf4bf4a2224758a12fc26f6a87aa95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } } peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } }
I20240501 14:09:05.415498 18368 ts_tablet_manager.cc:1398] T 78bc2976729d4670aab390265c3aca4a P eebf4bf4a2224758a12fc26f6a87aa95: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:05.431674 18376 raft_consensus.cc:482] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:05.431804 18376 raft_consensus.cc:504] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } }
I20240501 14:09:05.432354 18376 leader_election.cc:288] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eebf4bf4a2224758a12fc26f6a87aa95 (127.16.13.195:43181), 1d83861e0ee04f30a7c78a55a2ddd33e (127.16.13.193:42739)
I20240501 14:09:05.436012 18322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001cf18440d945b6824493c39b1d7487" candidate_uuid: "432945db49b84e008b12c241c76325ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" is_pre_election: true
I20240501 14:09:05.436033 18171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001cf18440d945b6824493c39b1d7487" candidate_uuid: "432945db49b84e008b12c241c76325ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" is_pre_election: true
I20240501 14:09:05.436156 18322 raft_consensus.cc:2454] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 432945db49b84e008b12c241c76325ed in term 0.
I20240501 14:09:05.436185 18171 raft_consensus.cc:2454] T 001cf18440d945b6824493c39b1d7487 P 1d83861e0ee04f30a7c78a55a2ddd33e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 432945db49b84e008b12c241c76325ed in term 0.
I20240501 14:09:05.436458 18210 leader_election.cc:302] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d83861e0ee04f30a7c78a55a2ddd33e, 432945db49b84e008b12c241c76325ed; no voters:
I20240501 14:09:05.436610 18376 raft_consensus.cc:2781] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:05.436691 18376 raft_consensus.cc:482] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:05.436728 18376 raft_consensus.cc:3037] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:05.437506 18376 raft_consensus.cc:504] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } }
I20240501 14:09:05.437783 18376 leader_election.cc:288] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [CANDIDATE]: Term 1 election: Requested vote from peers eebf4bf4a2224758a12fc26f6a87aa95 (127.16.13.195:43181), 1d83861e0ee04f30a7c78a55a2ddd33e (127.16.13.193:42739)
I20240501 14:09:05.438042 18322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001cf18440d945b6824493c39b1d7487" candidate_uuid: "432945db49b84e008b12c241c76325ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebf4bf4a2224758a12fc26f6a87aa95"
I20240501 14:09:05.438138 18322 raft_consensus.cc:3037] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:05.438896 18322 raft_consensus.cc:2454] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 432945db49b84e008b12c241c76325ed in term 1.
I20240501 14:09:05.439069 18171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001cf18440d945b6824493c39b1d7487" candidate_uuid: "432945db49b84e008b12c241c76325ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e"
I20240501 14:09:05.439162 18171 raft_consensus.cc:3037] T 001cf18440d945b6824493c39b1d7487 P 1d83861e0ee04f30a7c78a55a2ddd33e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:05.439196 18212 leader_election.cc:302] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 432945db49b84e008b12c241c76325ed, eebf4bf4a2224758a12fc26f6a87aa95; no voters:
I20240501 14:09:05.439347 18376 raft_consensus.cc:2781] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:05.439568 18376 raft_consensus.cc:686] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [term 1 LEADER]: Becoming Leader. State: Replica: 432945db49b84e008b12c241c76325ed, State: Running, Role: LEADER
I20240501 14:09:05.439713 18376 consensus_queue.cc:237] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } }
I20240501 14:09:05.440022 18171 raft_consensus.cc:2454] T 001cf18440d945b6824493c39b1d7487 P 1d83861e0ee04f30a7c78a55a2ddd33e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 432945db49b84e008b12c241c76325ed in term 1.
I20240501 14:09:05.441183 18068 catalog_manager.cc:5484] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 432945db49b84e008b12c241c76325ed (127.16.13.194). New cstate: current_term: 1 leader_uuid: "432945db49b84e008b12c241c76325ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:05.791440 18377 raft_consensus.cc:482] T 78bc2976729d4670aab390265c3aca4a P 1d83861e0ee04f30a7c78a55a2ddd33e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:05.791596 18377 raft_consensus.cc:504] T 78bc2976729d4670aab390265c3aca4a P 1d83861e0ee04f30a7c78a55a2ddd33e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } } peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } }
I20240501 14:09:05.792193 18377 leader_election.cc:288] T 78bc2976729d4670aab390265c3aca4a P 1d83861e0ee04f30a7c78a55a2ddd33e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 432945db49b84e008b12c241c76325ed (127.16.13.194:35635), eebf4bf4a2224758a12fc26f6a87aa95 (127.16.13.195:43181)
I20240501 14:09:05.792594 18376 raft_consensus.cc:482] T 78bc2976729d4670aab390265c3aca4a P 432945db49b84e008b12c241c76325ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:05.792694 18376 raft_consensus.cc:504] T 78bc2976729d4670aab390265c3aca4a P 432945db49b84e008b12c241c76325ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } } peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } }
I20240501 14:09:05.793066 18376 leader_election.cc:288] T 78bc2976729d4670aab390265c3aca4a P 432945db49b84e008b12c241c76325ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d83861e0ee04f30a7c78a55a2ddd33e (127.16.13.193:42739), eebf4bf4a2224758a12fc26f6a87aa95 (127.16.13.195:43181)
I20240501 14:09:05.793468 18322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bc2976729d4670aab390265c3aca4a" candidate_uuid: "432945db49b84e008b12c241c76325ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" is_pre_election: true
I20240501 14:09:05.793514 18171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bc2976729d4670aab390265c3aca4a" candidate_uuid: "432945db49b84e008b12c241c76325ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" is_pre_election: true
I20240501 14:09:05.793591 18322 raft_consensus.cc:2454] T 78bc2976729d4670aab390265c3aca4a P eebf4bf4a2224758a12fc26f6a87aa95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 432945db49b84e008b12c241c76325ed in term 0.
I20240501 14:09:05.793650 18171 raft_consensus.cc:2454] T 78bc2976729d4670aab390265c3aca4a P 1d83861e0ee04f30a7c78a55a2ddd33e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 432945db49b84e008b12c241c76325ed in term 0.
I20240501 14:09:05.793852 18212 leader_election.cc:302] T 78bc2976729d4670aab390265c3aca4a P 432945db49b84e008b12c241c76325ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 432945db49b84e008b12c241c76325ed, eebf4bf4a2224758a12fc26f6a87aa95; no voters:
I20240501 14:09:05.793998 18376 raft_consensus.cc:2781] T 78bc2976729d4670aab390265c3aca4a P 432945db49b84e008b12c241c76325ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:05.794062 18376 raft_consensus.cc:482] T 78bc2976729d4670aab390265c3aca4a P 432945db49b84e008b12c241c76325ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:05.794142 18376 raft_consensus.cc:3037] T 78bc2976729d4670aab390265c3aca4a P 432945db49b84e008b12c241c76325ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:05.795190 18376 raft_consensus.cc:504] T 78bc2976729d4670aab390265c3aca4a P 432945db49b84e008b12c241c76325ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } } peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } }
I20240501 14:09:05.795847 18171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bc2976729d4670aab390265c3aca4a" candidate_uuid: "432945db49b84e008b12c241c76325ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e"
I20240501 14:09:05.795948 18322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bc2976729d4670aab390265c3aca4a" candidate_uuid: "432945db49b84e008b12c241c76325ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebf4bf4a2224758a12fc26f6a87aa95"
I20240501 14:09:05.795995 18171 raft_consensus.cc:3037] T 78bc2976729d4670aab390265c3aca4a P 1d83861e0ee04f30a7c78a55a2ddd33e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:05.796047 18322 raft_consensus.cc:3037] T 78bc2976729d4670aab390265c3aca4a P eebf4bf4a2224758a12fc26f6a87aa95 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:05.796165 18247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bc2976729d4670aab390265c3aca4a" candidate_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "432945db49b84e008b12c241c76325ed" is_pre_election: true
I20240501 14:09:05.796304 18247 raft_consensus.cc:2379] T 78bc2976729d4670aab390265c3aca4a P 432945db49b84e008b12c241c76325ed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1d83861e0ee04f30a7c78a55a2ddd33e in current term 1: Already voted for candidate 432945db49b84e008b12c241c76325ed in this term.
I20240501 14:09:05.796824 18321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bc2976729d4670aab390265c3aca4a" candidate_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" is_pre_election: true
I20240501 14:09:05.796806 18376 leader_election.cc:288] T 78bc2976729d4670aab390265c3aca4a P 432945db49b84e008b12c241c76325ed [CANDIDATE]: Term 1 election: Requested vote from peers 1d83861e0ee04f30a7c78a55a2ddd33e (127.16.13.193:42739), eebf4bf4a2224758a12fc26f6a87aa95 (127.16.13.195:43181)
I20240501 14:09:05.797060 18171 raft_consensus.cc:2454] T 78bc2976729d4670aab390265c3aca4a P 1d83861e0ee04f30a7c78a55a2ddd33e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 432945db49b84e008b12c241c76325ed in term 1.
I20240501 14:09:05.797099 18322 raft_consensus.cc:2454] T 78bc2976729d4670aab390265c3aca4a P eebf4bf4a2224758a12fc26f6a87aa95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 432945db49b84e008b12c241c76325ed in term 1.
I20240501 14:09:05.797189 18136 leader_election.cc:302] T 78bc2976729d4670aab390265c3aca4a P 1d83861e0ee04f30a7c78a55a2ddd33e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d83861e0ee04f30a7c78a55a2ddd33e; no voters: 432945db49b84e008b12c241c76325ed, eebf4bf4a2224758a12fc26f6a87aa95
I20240501 14:09:05.797317 18210 leader_election.cc:302] T 78bc2976729d4670aab390265c3aca4a P 432945db49b84e008b12c241c76325ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d83861e0ee04f30a7c78a55a2ddd33e, 432945db49b84e008b12c241c76325ed; no voters:
I20240501 14:09:05.797369 18377 raft_consensus.cc:2726] T 78bc2976729d4670aab390265c3aca4a P 1d83861e0ee04f30a7c78a55a2ddd33e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:09:05.797487 18376 raft_consensus.cc:2781] T 78bc2976729d4670aab390265c3aca4a P 432945db49b84e008b12c241c76325ed [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:05.797585 18376 raft_consensus.cc:686] T 78bc2976729d4670aab390265c3aca4a P 432945db49b84e008b12c241c76325ed [term 1 LEADER]: Becoming Leader. State: Replica: 432945db49b84e008b12c241c76325ed, State: Running, Role: LEADER
I20240501 14:09:05.797722 18376 consensus_queue.cc:237] T 78bc2976729d4670aab390265c3aca4a P 432945db49b84e008b12c241c76325ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } } peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } }
I20240501 14:09:05.799101 18068 catalog_manager.cc:5484] T 78bc2976729d4670aab390265c3aca4a P 432945db49b84e008b12c241c76325ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 432945db49b84e008b12c241c76325ed (127.16.13.194). New cstate: current_term: 1 leader_uuid: "432945db49b84e008b12c241c76325ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:05.952606 18376 consensus_queue.cc:1035] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:05.955960 18382 consensus_queue.cc:1035] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:06.117769 18122 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:09:06.122023 18247 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "001cf18440d945b6824493c39b1d7487"
dest_uuid: "432945db49b84e008b12c241c76325ed"
mode: GRACEFUL
new_leader_uuid: "eebf4bf4a2224758a12fc26f6a87aa95"
from {username='slave'} at 127.0.0.1:59060
I20240501 14:09:06.122166 18247 raft_consensus.cc:595] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [term 1 LEADER]: Received request to transfer leadership to TS eebf4bf4a2224758a12fc26f6a87aa95
I20240501 14:09:06.122553 18122 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 14:09:06.122655 18122 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:09:06.256548 18382 consensus_queue.cc:1035] T 78bc2976729d4670aab390265c3aca4a P 432945db49b84e008b12c241c76325ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:06.259711 18376 consensus_queue.cc:1035] T 78bc2976729d4670aab390265c3aca4a P 432945db49b84e008b12c241c76325ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:06.416121 18376 raft_consensus.cc:979] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed: : Instructing follower eebf4bf4a2224758a12fc26f6a87aa95 to start an election
I20240501 14:09:06.416272 18376 raft_consensus.cc:1067] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [term 1 LEADER]: Signalling peer eebf4bf4a2224758a12fc26f6a87aa95 to start an election
I20240501 14:09:06.416777 18322 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "001cf18440d945b6824493c39b1d7487"
dest_uuid: "eebf4bf4a2224758a12fc26f6a87aa95"
from {username='slave'} at 127.0.0.1:51302
I20240501 14:09:06.416934 18322 raft_consensus.cc:482] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:06.416998 18322 raft_consensus.cc:3037] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:06.417896 18322 raft_consensus.cc:504] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } }
I20240501 14:09:06.418304 18322 leader_election.cc:288] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [CANDIDATE]: Term 2 election: Requested vote from peers 432945db49b84e008b12c241c76325ed (127.16.13.194:35635), 1d83861e0ee04f30a7c78a55a2ddd33e (127.16.13.193:42739)
I20240501 14:09:06.421734 18171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001cf18440d945b6824493c39b1d7487" candidate_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e"
I20240501 14:09:06.421850 18247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001cf18440d945b6824493c39b1d7487" candidate_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "432945db49b84e008b12c241c76325ed"
I20240501 14:09:06.421865 18171 raft_consensus.cc:3037] T 001cf18440d945b6824493c39b1d7487 P 1d83861e0ee04f30a7c78a55a2ddd33e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:06.421985 18247 raft_consensus.cc:3032] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:06.422036 18247 raft_consensus.cc:729] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 432945db49b84e008b12c241c76325ed, State: Running, Role: LEADER
I20240501 14:09:06.422194 18247 consensus_queue.cc:260] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } }
I20240501 14:09:06.422405 18247 raft_consensus.cc:3037] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:06.422823 18171 raft_consensus.cc:2454] T 001cf18440d945b6824493c39b1d7487 P 1d83861e0ee04f30a7c78a55a2ddd33e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eebf4bf4a2224758a12fc26f6a87aa95 in term 2.
I20240501 14:09:06.423090 18285 leader_election.cc:302] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d83861e0ee04f30a7c78a55a2ddd33e, eebf4bf4a2224758a12fc26f6a87aa95; no voters:
I20240501 14:09:06.423221 18247 raft_consensus.cc:2454] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eebf4bf4a2224758a12fc26f6a87aa95 in term 2.
I20240501 14:09:06.423260 18390 raft_consensus.cc:2781] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:06.423528 18390 raft_consensus.cc:686] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [term 2 LEADER]: Becoming Leader. State: Replica: eebf4bf4a2224758a12fc26f6a87aa95, State: Running, Role: LEADER
I20240501 14:09:06.423718 18390 consensus_queue.cc:237] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } }
I20240501 14:09:06.425117 18068 catalog_manager.cc:5484] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 reported cstate change: term changed from 1 to 2, leader changed from 432945db49b84e008b12c241c76325ed (127.16.13.194) to eebf4bf4a2224758a12fc26f6a87aa95 (127.16.13.195). New cstate: current_term: 2 leader_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:06.983218 18247 raft_consensus.cc:1261] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [term 2 FOLLOWER]: Refusing update from remote peer eebf4bf4a2224758a12fc26f6a87aa95: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:09:06.983690 18404 consensus_queue.cc:1035] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:06.986158 18171 raft_consensus.cc:1261] T 001cf18440d945b6824493c39b1d7487 P 1d83861e0ee04f30a7c78a55a2ddd33e [term 2 FOLLOWER]: Refusing update from remote peer eebf4bf4a2224758a12fc26f6a87aa95: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:09:06.986546 18406 consensus_queue.cc:1035] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:08.122943 18122 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:09:08.123399 18122 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:09:08.123473 18122 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:09:10.123761 18122 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:09:10.124200 18122 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:09:10.124269 18122 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:09:12.124536 18122 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:09:12.124902 18122 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:09:12.124965 18122 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:09:14.125233 18122 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:09:14.125598 18122 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:09:14.125670 18122 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:09:16.125981 18122 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:09:16.126590 18122 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:09:16.126685 18122 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:09:16.902294 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:16.905126 18415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:16.906103 18417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:16.906951 18414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:16.907680 16439 server_base.cc:1034] running on GCE node
I20240501 14:09:16.907866 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:09:16.907907 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:09:16.907970 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572556907970 us; error 0 us; skew 500 ppm
I20240501 14:09:16.909060 16439 webserver.cc:460] Webserver started at http://127.16.13.196:38141/ using document root <none> and password file <none>
I20240501 14:09:16.909225 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:16.909276 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:16.909343 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:16.909725 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714572529860089-16439-0/minicluster-data/ts-3-root/instance:
uuid: "93d058a124884d85b7b07865f94d3bc8"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-mgxc"
I20240501 14:09:16.911090 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 14:09:16.911985 18422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.912233 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:16.912302 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714572529860089-16439-0/minicluster-data/ts-3-root
uuid: "93d058a124884d85b7b07865f94d3bc8"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-mgxc"
I20240501 14:09:16.912351 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714572529860089-16439-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714572529860089-16439-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714572529860089-16439-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:16.926136 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:16.926463 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:16.926723 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:16.927155 16439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:16.927191 16439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.927222 16439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:16.927237 16439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.931704 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.196:45229
I20240501 14:09:16.931764 18485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.196:45229 every 8 connection(s)
I20240501 14:09:16.936106 18486 heartbeater.cc:347] Connected to a master server at 127.16.13.254:42829
I20240501 14:09:16.936205 18486 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:16.936410 18486 heartbeater.cc:511] Master 127.16.13.254:42829 requested a full tablet report, sending...
I20240501 14:09:16.936944 18068 ts_manager.cc:194] Registered new tserver with Master: 93d058a124884d85b7b07865f94d3bc8 (127.16.13.196:45229)
I20240501 14:09:16.937441 18068 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57726
I20240501 14:09:17.133105 18322 consensus_queue.cc:237] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } } peers { permanent_uuid: "93d058a124884d85b7b07865f94d3bc8" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 45229 } attrs { promote: true } }
I20240501 14:09:17.134605 18247 raft_consensus.cc:1261] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [term 2 FOLLOWER]: Refusing update from remote peer eebf4bf4a2224758a12fc26f6a87aa95: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 14:09:17.134899 18170 raft_consensus.cc:1261] T 001cf18440d945b6824493c39b1d7487 P 1d83861e0ee04f30a7c78a55a2ddd33e [term 2 FOLLOWER]: Refusing update from remote peer eebf4bf4a2224758a12fc26f6a87aa95: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 14:09:17.135000 18493 consensus_queue.cc:1035] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:17.135288 18494 consensus_queue.cc:1035] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:17.136737 18412 raft_consensus.cc:2932] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 93d058a124884d85b7b07865f94d3bc8 (127.16.13.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } } peers { permanent_uuid: "93d058a124884d85b7b07865f94d3bc8" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 45229 } attrs { promote: true } } }
W20240501 14:09:17.139178 18288 consensus_peers.cc:485] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 -> Peer 93d058a124884d85b7b07865f94d3bc8 (127.16.13.196:45229): Couldn't send request to peer 93d058a124884d85b7b07865f94d3bc8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 001cf18440d945b6824493c39b1d7487. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:17.139369 18170 raft_consensus.cc:2932] T 001cf18440d945b6824493c39b1d7487 P 1d83861e0ee04f30a7c78a55a2ddd33e [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 93d058a124884d85b7b07865f94d3bc8 (127.16.13.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } } peers { permanent_uuid: "93d058a124884d85b7b07865f94d3bc8" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 45229 } attrs { promote: true } } }
I20240501 14:09:17.139811 18067 catalog_manager.cc:5484] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 reported cstate change: config changed from index -1 to 3, NON_VOTER 93d058a124884d85b7b07865f94d3bc8 (127.16.13.196) added. New cstate: current_term: 2 leader_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93d058a124884d85b7b07865f94d3bc8" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 45229 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:17.140107 18247 raft_consensus.cc:2932] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 93d058a124884d85b7b07865f94d3bc8 (127.16.13.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } } peers { permanent_uuid: "93d058a124884d85b7b07865f94d3bc8" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 45229 } attrs { promote: true } } }
I20240501 14:09:17.544085 18504 ts_tablet_manager.cc:902] T 001cf18440d945b6824493c39b1d7487 P 93d058a124884d85b7b07865f94d3bc8: Initiating tablet copy from peer eebf4bf4a2224758a12fc26f6a87aa95 (127.16.13.195:43181)
I20240501 14:09:17.544898 18504 tablet_copy_client.cc:308] T 001cf18440d945b6824493c39b1d7487 P 93d058a124884d85b7b07865f94d3bc8: tablet copy: Beginning tablet copy session from remote peer at address 127.16.13.195:43181
I20240501 14:09:17.548669 18332 tablet_copy_service.cc:140] P eebf4bf4a2224758a12fc26f6a87aa95: Received BeginTabletCopySession request for tablet 001cf18440d945b6824493c39b1d7487 from peer 93d058a124884d85b7b07865f94d3bc8 ({username='slave'} at 127.0.0.1:48352)
I20240501 14:09:17.548784 18332 tablet_copy_service.cc:161] P eebf4bf4a2224758a12fc26f6a87aa95: Beginning new tablet copy session on tablet 001cf18440d945b6824493c39b1d7487 from peer 93d058a124884d85b7b07865f94d3bc8 at {username='slave'} at 127.0.0.1:48352: session id = 93d058a124884d85b7b07865f94d3bc8-001cf18440d945b6824493c39b1d7487
I20240501 14:09:17.550247 18332 tablet_copy_source_session.cc:205] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:17.551517 18504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 001cf18440d945b6824493c39b1d7487. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:17.553891 18504 tablet_copy_client.cc:783] T 001cf18440d945b6824493c39b1d7487 P 93d058a124884d85b7b07865f94d3bc8: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:17.554059 18504 tablet_copy_client.cc:647] T 001cf18440d945b6824493c39b1d7487 P 93d058a124884d85b7b07865f94d3bc8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:17.556219 18504 tablet_copy_client.cc:523] T 001cf18440d945b6824493c39b1d7487 P 93d058a124884d85b7b07865f94d3bc8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:17.557854 18504 tablet_bootstrap.cc:492] T 001cf18440d945b6824493c39b1d7487 P 93d058a124884d85b7b07865f94d3bc8: Bootstrap starting.
I20240501 14:09:17.561779 18504 tablet_bootstrap.cc:492] T 001cf18440d945b6824493c39b1d7487 P 93d058a124884d85b7b07865f94d3bc8: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:17.562011 18504 tablet_bootstrap.cc:492] T 001cf18440d945b6824493c39b1d7487 P 93d058a124884d85b7b07865f94d3bc8: Bootstrap complete.
I20240501 14:09:17.562162 18504 ts_tablet_manager.cc:1367] T 001cf18440d945b6824493c39b1d7487 P 93d058a124884d85b7b07865f94d3bc8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240501 14:09:17.562556 18504 raft_consensus.cc:348] T 001cf18440d945b6824493c39b1d7487 P 93d058a124884d85b7b07865f94d3bc8 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } } peers { permanent_uuid: "93d058a124884d85b7b07865f94d3bc8" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 45229 } attrs { promote: true } }
I20240501 14:09:17.562685 18504 raft_consensus.cc:729] T 001cf18440d945b6824493c39b1d7487 P 93d058a124884d85b7b07865f94d3bc8 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 93d058a124884d85b7b07865f94d3bc8, State: Initialized, Role: LEARNER
I20240501 14:09:17.562767 18504 consensus_queue.cc:260] T 001cf18440d945b6824493c39b1d7487 P 93d058a124884d85b7b07865f94d3bc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } } peers { permanent_uuid: "93d058a124884d85b7b07865f94d3bc8" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 45229 } attrs { promote: true } }
I20240501 14:09:17.563117 18504 ts_tablet_manager.cc:1398] T 001cf18440d945b6824493c39b1d7487 P 93d058a124884d85b7b07865f94d3bc8: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 14:09:17.563145 18486 heartbeater.cc:503] Master 127.16.13.254:42829 was elected leader, sending a full tablet report...
I20240501 14:09:17.563778 18332 tablet_copy_service.cc:342] P eebf4bf4a2224758a12fc26f6a87aa95: Request end of tablet copy session 93d058a124884d85b7b07865f94d3bc8-001cf18440d945b6824493c39b1d7487 received from {username='slave'} at 127.0.0.1:48352
I20240501 14:09:17.563863 18332 tablet_copy_service.cc:433] P eebf4bf4a2224758a12fc26f6a87aa95: ending tablet copy session 93d058a124884d85b7b07865f94d3bc8-001cf18440d945b6824493c39b1d7487 on tablet 001cf18440d945b6824493c39b1d7487 with peer 93d058a124884d85b7b07865f94d3bc8
I20240501 14:09:18.042783 18460 raft_consensus.cc:1203] T 001cf18440d945b6824493c39b1d7487 P 93d058a124884d85b7b07865f94d3bc8 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240501 14:09:18.126873 18122 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:09:18.127198 18122 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:09:18.127267 18122 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:09:18.554857 18513 raft_consensus.cc:1050] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95: attempting to promote NON_VOTER 93d058a124884d85b7b07865f94d3bc8 to VOTER
I20240501 14:09:18.555406 18513 consensus_queue.cc:237] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } } peers { permanent_uuid: "93d058a124884d85b7b07865f94d3bc8" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 45229 } attrs { promote: false } }
I20240501 14:09:18.556685 18460 raft_consensus.cc:1261] T 001cf18440d945b6824493c39b1d7487 P 93d058a124884d85b7b07865f94d3bc8 [term 2 LEARNER]: Refusing update from remote peer eebf4bf4a2224758a12fc26f6a87aa95: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:09:18.556967 18247 raft_consensus.cc:1261] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [term 2 FOLLOWER]: Refusing update from remote peer eebf4bf4a2224758a12fc26f6a87aa95: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:09:18.557232 18170 raft_consensus.cc:1261] T 001cf18440d945b6824493c39b1d7487 P 1d83861e0ee04f30a7c78a55a2ddd33e [term 2 FOLLOWER]: Refusing update from remote peer eebf4bf4a2224758a12fc26f6a87aa95: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:09:18.557191 18513 consensus_queue.cc:1035] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93d058a124884d85b7b07865f94d3bc8" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 45229 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:18.557353 18514 consensus_queue.cc:1035] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:18.557585 18516 consensus_queue.cc:1035] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:18.559578 18513 raft_consensus.cc:2932] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 93d058a124884d85b7b07865f94d3bc8 (127.16.13.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } } peers { permanent_uuid: "93d058a124884d85b7b07865f94d3bc8" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 45229 } attrs { promote: false } } }
I20240501 14:09:18.559978 18247 raft_consensus.cc:2932] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 93d058a124884d85b7b07865f94d3bc8 (127.16.13.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } } peers { permanent_uuid: "93d058a124884d85b7b07865f94d3bc8" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 45229 } attrs { promote: false } } }
I20240501 14:09:18.561340 18170 raft_consensus.cc:2932] T 001cf18440d945b6824493c39b1d7487 P 1d83861e0ee04f30a7c78a55a2ddd33e [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 93d058a124884d85b7b07865f94d3bc8 (127.16.13.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } } peers { permanent_uuid: "93d058a124884d85b7b07865f94d3bc8" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 45229 } attrs { promote: false } } }
I20240501 14:09:18.562021 18068 catalog_manager.cc:5484] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed reported cstate change: config changed from index 3 to 4, 93d058a124884d85b7b07865f94d3bc8 (127.16.13.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } } peers { permanent_uuid: "93d058a124884d85b7b07865f94d3bc8" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 45229 } attrs { promote: false } } }
I20240501 14:09:18.563382 18460 raft_consensus.cc:2932] T 001cf18440d945b6824493c39b1d7487 P 93d058a124884d85b7b07865f94d3bc8 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 93d058a124884d85b7b07865f94d3bc8 (127.16.13.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } } peers { permanent_uuid: "432945db49b84e008b12c241c76325ed" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } } peers { permanent_uuid: "93d058a124884d85b7b07865f94d3bc8" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 45229 } attrs { promote: false } } }
I20240501 14:09:18.567723 18321 consensus_queue.cc:237] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } } peers { permanent_uuid: "93d058a124884d85b7b07865f94d3bc8" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 45229 } attrs { promote: false } }
I20240501 14:09:18.568706 18460 raft_consensus.cc:1261] T 001cf18440d945b6824493c39b1d7487 P 93d058a124884d85b7b07865f94d3bc8 [term 2 FOLLOWER]: Refusing update from remote peer eebf4bf4a2224758a12fc26f6a87aa95: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:09:18.568779 18170 raft_consensus.cc:1261] T 001cf18440d945b6824493c39b1d7487 P 1d83861e0ee04f30a7c78a55a2ddd33e [term 2 FOLLOWER]: Refusing update from remote peer eebf4bf4a2224758a12fc26f6a87aa95: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:09:18.569015 18517 consensus_queue.cc:1035] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93d058a124884d85b7b07865f94d3bc8" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 45229 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:18.569154 18513 consensus_queue.cc:1035] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:18.570477 18516 raft_consensus.cc:2932] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 432945db49b84e008b12c241c76325ed (127.16.13.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } } peers { permanent_uuid: "93d058a124884d85b7b07865f94d3bc8" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 45229 } attrs { promote: false } } }
I20240501 14:09:18.570874 18170 raft_consensus.cc:2932] T 001cf18440d945b6824493c39b1d7487 P 1d83861e0ee04f30a7c78a55a2ddd33e [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 432945db49b84e008b12c241c76325ed (127.16.13.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } } peers { permanent_uuid: "93d058a124884d85b7b07865f94d3bc8" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 45229 } attrs { promote: false } } }
I20240501 14:09:18.570874 18460 raft_consensus.cc:2932] T 001cf18440d945b6824493c39b1d7487 P 93d058a124884d85b7b07865f94d3bc8 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 432945db49b84e008b12c241c76325ed (127.16.13.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } } peers { permanent_uuid: "93d058a124884d85b7b07865f94d3bc8" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 45229 } attrs { promote: false } } }
I20240501 14:09:18.572130 18054 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 001cf18440d945b6824493c39b1d7487 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:18.573093 18067 catalog_manager.cc:5484] T 001cf18440d945b6824493c39b1d7487 P 1d83861e0ee04f30a7c78a55a2ddd33e reported cstate change: config changed from index 4 to 5, VOTER 432945db49b84e008b12c241c76325ed (127.16.13.194) evicted. New cstate: current_term: 2 leader_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eebf4bf4a2224758a12fc26f6a87aa95" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43181 } } peers { permanent_uuid: "1d83861e0ee04f30a7c78a55a2ddd33e" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 42739 } } peers { permanent_uuid: "93d058a124884d85b7b07865f94d3bc8" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 45229 } attrs { promote: false } } }
I20240501 14:09:18.575301 18236 tablet_service.cc:1514] Processing DeleteTablet for tablet 001cf18440d945b6824493c39b1d7487 with delete_type TABLET_DATA_TOMBSTONED (TS 432945db49b84e008b12c241c76325ed not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:59052
I20240501 14:09:18.575713 18527 tablet_replica.cc:330] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed: stopping tablet replica
I20240501 14:09:18.575829 18527 raft_consensus.cc:2229] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:18.575961 18527 raft_consensus.cc:2258] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:18.576421 18527 ts_tablet_manager.cc:1875] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:18.578002 18527 ts_tablet_manager.cc:1888] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240501 14:09:18.578071 18527 log.cc:1196] T 001cf18440d945b6824493c39b1d7487 P 432945db49b84e008b12c241c76325ed: Deleting WAL directory at /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714572529860089-16439-0/minicluster-data/ts-1-root/wals/001cf18440d945b6824493c39b1d7487
I20240501 14:09:18.578516 18053 catalog_manager.cc:4830] TS 432945db49b84e008b12c241c76325ed (127.16.13.194:35635): tablet 001cf18440d945b6824493c39b1d7487 (table test-workload [id=9df3269fbabd4e04aa4aa4e32b8797fc]) successfully deleted
I20240501 14:09:20.127547 18122 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:09:20.128007 18122 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:09:20.128077 18122 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:09:22.128327 18122 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:09:22.128757 18122 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:09:22.128834 18122 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:09:24.129153 18122 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:09:24.129581 18122 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:09:24.129653 18122 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:09:26.129938 18122 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:09:26.130398 18122 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:09:26.130471 18122 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:09:28.130787 18122 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:09:28.131325 18122 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:09:28.131430 18122 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:09:30.131740 18122 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:09:30.132193 18122 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:09:30.132268 18122 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:09:30.758445 16439 tablet_server.cc:178] TabletServer@127.16.13.193:0 shutting down...
I20240501 14:09:30.762449 16439 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:09:30.762642 16439 tablet_replica.cc:330] T 001cf18440d945b6824493c39b1d7487 P 1d83861e0ee04f30a7c78a55a2ddd33e: stopping tablet replica
I20240501 14:09:30.762759 16439 raft_consensus.cc:2229] T 001cf18440d945b6824493c39b1d7487 P 1d83861e0ee04f30a7c78a55a2ddd33e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:30.762895 16439 raft_consensus.cc:2258] T 001cf18440d945b6824493c39b1d7487 P 1d83861e0ee04f30a7c78a55a2ddd33e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:30.763372 16439 tablet_replica.cc:330] T 78bc2976729d4670aab390265c3aca4a P 1d83861e0ee04f30a7c78a55a2ddd33e: stopping tablet replica
I20240501 14:09:30.763454 16439 raft_consensus.cc:2229] T 78bc2976729d4670aab390265c3aca4a P 1d83861e0ee04f30a7c78a55a2ddd33e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:30.763533 16439 raft_consensus.cc:2258] T 78bc2976729d4670aab390265c3aca4a P 1d83861e0ee04f30a7c78a55a2ddd33e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:30.776223 16439 tablet_server.cc:194] TabletServer@127.16.13.193:0 shutdown complete.
I20240501 14:09:30.778270 16439 tablet_server.cc:178] TabletServer@127.16.13.194:0 shutting down...
I20240501 14:09:30.782632 16439 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:09:30.782820 16439 tablet_replica.cc:330] T 78bc2976729d4670aab390265c3aca4a P 432945db49b84e008b12c241c76325ed: stopping tablet replica
I20240501 14:09:30.782909 16439 raft_consensus.cc:2229] T 78bc2976729d4670aab390265c3aca4a P 432945db49b84e008b12c241c76325ed [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:30.783073 16439 raft_consensus.cc:2258] T 78bc2976729d4670aab390265c3aca4a P 432945db49b84e008b12c241c76325ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:30.796691 16439 tablet_server.cc:194] TabletServer@127.16.13.194:0 shutdown complete.
I20240501 14:09:30.798621 16439 tablet_server.cc:178] TabletServer@127.16.13.195:0 shutting down...
I20240501 14:09:30.802637 16439 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:09:30.802803 16439 tablet_replica.cc:330] T 78bc2976729d4670aab390265c3aca4a P eebf4bf4a2224758a12fc26f6a87aa95: stopping tablet replica
I20240501 14:09:30.802904 16439 raft_consensus.cc:2229] T 78bc2976729d4670aab390265c3aca4a P eebf4bf4a2224758a12fc26f6a87aa95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:30.803012 16439 raft_consensus.cc:2258] T 78bc2976729d4670aab390265c3aca4a P eebf4bf4a2224758a12fc26f6a87aa95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:30.803413 16439 tablet_replica.cc:330] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95: stopping tablet replica
I20240501 14:09:30.803503 16439 raft_consensus.cc:2229] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [term 2 LEADER]: Raft consensus shutting down.
I20240501 14:09:30.803668 16439 raft_consensus.cc:2258] T 001cf18440d945b6824493c39b1d7487 P eebf4bf4a2224758a12fc26f6a87aa95 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:30.817126 16439 tablet_server.cc:194] TabletServer@127.16.13.195:0 shutdown complete.
I20240501 14:09:30.819075 16439 tablet_server.cc:178] TabletServer@127.16.13.196:0 shutting down...
I20240501 14:09:30.823127 16439 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:09:30.823289 16439 tablet_replica.cc:330] T 001cf18440d945b6824493c39b1d7487 P 93d058a124884d85b7b07865f94d3bc8: stopping tablet replica
I20240501 14:09:30.823391 16439 raft_consensus.cc:2229] T 001cf18440d945b6824493c39b1d7487 P 93d058a124884d85b7b07865f94d3bc8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:30.823537 16439 raft_consensus.cc:2258] T 001cf18440d945b6824493c39b1d7487 P 93d058a124884d85b7b07865f94d3bc8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:30.835701 16439 tablet_server.cc:194] TabletServer@127.16.13.196:0 shutdown complete.
I20240501 14:09:30.837901 16439 master.cc:537] Master@127.16.13.254:42829 shutting down...
I20240501 14:09:30.846952 16439 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 77b8eff2377b453b94d0789d717c0a05 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:30.847129 16439 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 77b8eff2377b453b94d0789d717c0a05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:30.847200 16439 tablet_replica.cc:330] T 00000000000000000000000000000000 P 77b8eff2377b453b94d0789d717c0a05: stopping tablet replica
I20240501 14:09:30.859771 16439 master.cc:552] Master@127.16.13.254:42829 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (26780 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20240501 14:09:30.867249 16439 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.13.254:36501
I20240501 14:09:30.867550 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:30.869952 18531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:30.870710 18533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:30.871150 18530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:30.872131 16439 server_base.cc:1034] running on GCE node
I20240501 14:09:30.872272 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:09:30.872311 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:09:30.872326 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572570872326 us; error 0 us; skew 500 ppm
I20240501 14:09:30.876616 16439 webserver.cc:460] Webserver started at http://127.16.13.254:37001/ using document root <none> and password file <none>
I20240501 14:09:30.876770 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:30.876832 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:30.876899 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:30.877351 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714572529860089-16439-0/minicluster-data/master-0-root/instance:
uuid: "1a383e994a4d4fc4973a45708c6425f0"
format_stamp: "Formatted at 2024-05-01 14:09:30 on dist-test-slave-mgxc"
I20240501 14:09:30.878811 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:30.879750 18538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:30.880067 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:30.880156 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714572529860089-16439-0/minicluster-data/master-0-root
uuid: "1a383e994a4d4fc4973a45708c6425f0"
format_stamp: "Formatted at 2024-05-01 14:09:30 on dist-test-slave-mgxc"
I20240501 14:09:30.880234 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714572529860089-16439-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714572529860089-16439-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714572529860089-16439-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:30.892674 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:30.893002 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:30.896852 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.254:36501
I20240501 14:09:30.899152 18590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.254:36501 every 8 connection(s)
I20240501 14:09:30.899574 18591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.901365 18591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a383e994a4d4fc4973a45708c6425f0: Bootstrap starting.
I20240501 14:09:30.902083 18591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a383e994a4d4fc4973a45708c6425f0: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.903007 18591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a383e994a4d4fc4973a45708c6425f0: No bootstrap required, opened a new log
I20240501 14:09:30.903369 18591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a383e994a4d4fc4973a45708c6425f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a383e994a4d4fc4973a45708c6425f0" member_type: VOTER }
I20240501 14:09:30.903453 18591 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a383e994a4d4fc4973a45708c6425f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.903475 18591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a383e994a4d4fc4973a45708c6425f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a383e994a4d4fc4973a45708c6425f0, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.903626 18591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a383e994a4d4fc4973a45708c6425f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a383e994a4d4fc4973a45708c6425f0" member_type: VOTER }
I20240501 14:09:30.903690 18591 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a383e994a4d4fc4973a45708c6425f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:09:30.903728 18591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a383e994a4d4fc4973a45708c6425f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:09:30.903779 18591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a383e994a4d4fc4973a45708c6425f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.904453 18591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a383e994a4d4fc4973a45708c6425f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a383e994a4d4fc4973a45708c6425f0" member_type: VOTER }
I20240501 14:09:30.904559 18591 leader_election.cc:302] T 00000000000000000000000000000000 P 1a383e994a4d4fc4973a45708c6425f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a383e994a4d4fc4973a45708c6425f0; no voters:
I20240501 14:09:30.904726 18591 leader_election.cc:288] T 00000000000000000000000000000000 P 1a383e994a4d4fc4973a45708c6425f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:09:30.904862 18594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a383e994a4d4fc4973a45708c6425f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:30.905076 18594 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a383e994a4d4fc4973a45708c6425f0 [term 1 LEADER]: Becoming Leader. State: Replica: 1a383e994a4d4fc4973a45708c6425f0, State: Running, Role: LEADER
I20240501 14:09:30.905223 18591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a383e994a4d4fc4973a45708c6425f0 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:09:30.905226 18594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a383e994a4d4fc4973a45708c6425f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a383e994a4d4fc4973a45708c6425f0" member_type: VOTER }
I20240501 14:09:30.905685 18595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a383e994a4d4fc4973a45708c6425f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a383e994a4d4fc4973a45708c6425f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a383e994a4d4fc4973a45708c6425f0" member_type: VOTER } }
I20240501 14:09:30.905774 18595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a383e994a4d4fc4973a45708c6425f0 [sys.catalog]: This master's current role is: LEADER
I20240501 14:09:30.905723 18596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a383e994a4d4fc4973a45708c6425f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a383e994a4d4fc4973a45708c6425f0. Latest consensus state: current_term: 1 leader_uuid: "1a383e994a4d4fc4973a45708c6425f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a383e994a4d4fc4973a45708c6425f0" member_type: VOTER } }
I20240501 14:09:30.905819 18596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a383e994a4d4fc4973a45708c6425f0 [sys.catalog]: This master's current role is: LEADER
I20240501 14:09:30.906026 18599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:09:30.907047 18599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:09:30.907279 16439 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 14:09:30.908864 18599 catalog_manager.cc:1347] Generated new cluster ID: 5ccb2359dfaf421da1bb9b7bca4c4e2e
I20240501 14:09:30.908910 18599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:09:30.917788 18599 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:09:30.918278 18599 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:09:30.929337 18599 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1a383e994a4d4fc4973a45708c6425f0: Generated new TSK 0
I20240501 14:09:30.929491 18599 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:09:30.939178 16439 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 6.3208e-05s
I20240501 14:09:43.017544 16439 master.cc:537] Master@127.16.13.254:36501 shutting down...
I20240501 14:09:43.021306 16439 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1a383e994a4d4fc4973a45708c6425f0 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:43.021507 16439 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1a383e994a4d4fc4973a45708c6425f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:43.021584 16439 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1a383e994a4d4fc4973a45708c6425f0: stopping tablet replica
I20240501 14:09:43.033610 16439 master.cc:552] Master@127.16.13.254:36501 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12170 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20240501 14:09:43.037829 16439 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.13.254:38013
I20240501 14:09:43.038156 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:43.040668 18616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:43.041569 18614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:43.042622 16439 server_base.cc:1034] running on GCE node
W20240501 14:09:43.043056 18613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:43.043263 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:09:43.043305 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:09:43.043323 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572583043322 us; error 0 us; skew 500 ppm
I20240501 14:09:43.047402 16439 webserver.cc:460] Webserver started at http://127.16.13.254:39881/ using document root <none> and password file <none>
I20240501 14:09:43.047537 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:43.047580 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:43.047641 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:43.048046 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714572529860089-16439-0/minicluster-data/master-0-root/instance:
uuid: "519862f6785d42ec834085dbbcef9c75"
format_stamp: "Formatted at 2024-05-01 14:09:43 on dist-test-slave-mgxc"
I20240501 14:09:43.049413 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:43.050232 18621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:43.050501 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:43.050561 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714572529860089-16439-0/minicluster-data/master-0-root
uuid: "519862f6785d42ec834085dbbcef9c75"
format_stamp: "Formatted at 2024-05-01 14:09:43 on dist-test-slave-mgxc"
I20240501 14:09:43.050614 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714572529860089-16439-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714572529860089-16439-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714572529860089-16439-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:43.093997 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:43.094437 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:43.098963 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.254:38013
I20240501 14:09:43.099653 18673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.254:38013 every 8 connection(s)
I20240501 14:09:43.100282 18674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.102164 18674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 519862f6785d42ec834085dbbcef9c75: Bootstrap starting.
I20240501 14:09:43.102927 18674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 519862f6785d42ec834085dbbcef9c75: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.104066 18674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 519862f6785d42ec834085dbbcef9c75: No bootstrap required, opened a new log
I20240501 14:09:43.104434 18674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 519862f6785d42ec834085dbbcef9c75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519862f6785d42ec834085dbbcef9c75" member_type: VOTER }
I20240501 14:09:43.104519 18674 raft_consensus.cc:374] T 00000000000000000000000000000000 P 519862f6785d42ec834085dbbcef9c75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.104542 18674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 519862f6785d42ec834085dbbcef9c75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519862f6785d42ec834085dbbcef9c75, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.104656 18674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 519862f6785d42ec834085dbbcef9c75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519862f6785d42ec834085dbbcef9c75" member_type: VOTER }
I20240501 14:09:43.104733 18674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 519862f6785d42ec834085dbbcef9c75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:09:43.104758 18674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 519862f6785d42ec834085dbbcef9c75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:09:43.104785 18674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 519862f6785d42ec834085dbbcef9c75 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.105440 18674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 519862f6785d42ec834085dbbcef9c75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519862f6785d42ec834085dbbcef9c75" member_type: VOTER }
I20240501 14:09:43.105547 18674 leader_election.cc:302] T 00000000000000000000000000000000 P 519862f6785d42ec834085dbbcef9c75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 519862f6785d42ec834085dbbcef9c75; no voters:
I20240501 14:09:43.105703 18674 leader_election.cc:288] T 00000000000000000000000000000000 P 519862f6785d42ec834085dbbcef9c75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:09:43.105845 18677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 519862f6785d42ec834085dbbcef9c75 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:43.106089 18674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 519862f6785d42ec834085dbbcef9c75 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:09:43.106132 18677 raft_consensus.cc:686] T 00000000000000000000000000000000 P 519862f6785d42ec834085dbbcef9c75 [term 1 LEADER]: Becoming Leader. State: Replica: 519862f6785d42ec834085dbbcef9c75, State: Running, Role: LEADER
I20240501 14:09:43.106305 18677 consensus_queue.cc:237] T 00000000000000000000000000000000 P 519862f6785d42ec834085dbbcef9c75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519862f6785d42ec834085dbbcef9c75" member_type: VOTER }
I20240501 14:09:43.106819 18679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519862f6785d42ec834085dbbcef9c75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 519862f6785d42ec834085dbbcef9c75. Latest consensus state: current_term: 1 leader_uuid: "519862f6785d42ec834085dbbcef9c75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519862f6785d42ec834085dbbcef9c75" member_type: VOTER } }
I20240501 14:09:43.106925 18679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519862f6785d42ec834085dbbcef9c75 [sys.catalog]: This master's current role is: LEADER
I20240501 14:09:43.107139 18678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519862f6785d42ec834085dbbcef9c75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "519862f6785d42ec834085dbbcef9c75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519862f6785d42ec834085dbbcef9c75" member_type: VOTER } }
I20240501 14:09:43.107231 18678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519862f6785d42ec834085dbbcef9c75 [sys.catalog]: This master's current role is: LEADER
I20240501 14:09:43.107583 18686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:09:43.108429 18686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:09:43.108592 16439 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 14:09:43.110345 18686 catalog_manager.cc:1347] Generated new cluster ID: f3e6e2e6ae87497aae8a7fe7ea7c1245
I20240501 14:09:43.110407 18686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:09:43.119031 18686 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:09:43.119632 18686 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:09:43.124575 18686 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 519862f6785d42ec834085dbbcef9c75: Generated new TSK 0
I20240501 14:09:43.124758 18686 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:09:43.141144 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:43.143821 18695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:43.144330 18696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:43.145020 18698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:43.145936 16439 server_base.cc:1034] running on GCE node
I20240501 14:09:43.146106 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:09:43.146140 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:09:43.146157 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572583146156 us; error 0 us; skew 500 ppm
I20240501 14:09:43.147171 16439 webserver.cc:460] Webserver started at http://127.16.13.193:41945/ using document root <none> and password file <none>
I20240501 14:09:43.147329 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:43.147375 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:43.147457 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:43.147877 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714572529860089-16439-0/minicluster-data/ts-0-root/instance:
uuid: "03ddbb61723f4682ab47152fad1aa626"
format_stamp: "Formatted at 2024-05-01 14:09:43 on dist-test-slave-mgxc"
I20240501 14:09:43.149408 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:43.150254 18703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:43.150485 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:43.150553 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714572529860089-16439-0/minicluster-data/ts-0-root
uuid: "03ddbb61723f4682ab47152fad1aa626"
format_stamp: "Formatted at 2024-05-01 14:09:43 on dist-test-slave-mgxc"
I20240501 14:09:43.150620 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714572529860089-16439-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714572529860089-16439-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714572529860089-16439-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:43.165109 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:43.165518 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:43.165800 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:43.166250 16439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:43.166368 16439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:43.166450 16439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:43.166484 16439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:43.170444 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.193:40237
I20240501 14:09:43.170495 18766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.193:40237 every 8 connection(s)
I20240501 14:09:43.171216 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:43.173661 18771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:43.174501 18774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:43.175285 18772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:43.175747 18767 heartbeater.cc:347] Connected to a master server at 127.16.13.254:38013
I20240501 14:09:43.175851 18767 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:43.176074 16439 server_base.cc:1034] running on GCE node
I20240501 14:09:43.176118 18767 heartbeater.cc:511] Master 127.16.13.254:38013 requested a full tablet report, sending...
I20240501 14:09:43.176256 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:09:43.176298 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:09:43.176319 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572583176318 us; error 0 us; skew 500 ppm
I20240501 14:09:43.176754 18638 ts_manager.cc:194] Registered new tserver with Master: 03ddbb61723f4682ab47152fad1aa626 (127.16.13.193:40237)
I20240501 14:09:43.177250 16439 webserver.cc:460] Webserver started at http://127.16.13.194:34543/ using document root <none> and password file <none>
I20240501 14:09:43.177393 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:43.177438 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:43.177513 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:43.177655 18638 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59536
I20240501 14:09:43.177918 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714572529860089-16439-0/minicluster-data/ts-1-root/instance:
uuid: "d55f6650f1934528aaa9d16f2785dafb"
format_stamp: "Formatted at 2024-05-01 14:09:43 on dist-test-slave-mgxc"
I20240501 14:09:43.179277 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:43.180081 18779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:43.180294 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:43.180356 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714572529860089-16439-0/minicluster-data/ts-1-root
uuid: "d55f6650f1934528aaa9d16f2785dafb"
format_stamp: "Formatted at 2024-05-01 14:09:43 on dist-test-slave-mgxc"
I20240501 14:09:43.180424 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714572529860089-16439-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714572529860089-16439-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714572529860089-16439-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:43.205291 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:43.205663 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:43.205988 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:43.206507 16439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:43.206548 16439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:43.206597 16439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:43.206630 16439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:43.210775 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.194:42211
I20240501 14:09:43.210841 18842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.194:42211 every 8 connection(s)
I20240501 14:09:43.211532 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:43.214156 18846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:43.214895 16439 server_base.cc:1034] running on GCE node
W20240501 14:09:43.215663 18849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:43.215754 18843 heartbeater.cc:347] Connected to a master server at 127.16.13.254:38013
I20240501 14:09:43.215845 18843 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:43.216095 18843 heartbeater.cc:511] Master 127.16.13.254:38013 requested a full tablet report, sending...
W20240501 14:09:43.216395 18847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:43.216650 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:09:43.216696 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:09:43.216715 18638 ts_manager.cc:194] Registered new tserver with Master: d55f6650f1934528aaa9d16f2785dafb (127.16.13.194:42211)
I20240501 14:09:43.216722 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572583216721 us; error 0 us; skew 500 ppm
I20240501 14:09:43.217244 18638 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59544
I20240501 14:09:43.217878 16439 webserver.cc:460] Webserver started at http://127.16.13.195:46523/ using document root <none> and password file <none>
I20240501 14:09:43.218063 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:43.218134 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:43.218200 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:43.218567 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714572529860089-16439-0/minicluster-data/ts-2-root/instance:
uuid: "a67cb959fb7349c09d3b52e121b5e8e7"
format_stamp: "Formatted at 2024-05-01 14:09:43 on dist-test-slave-mgxc"
I20240501 14:09:43.220082 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:43.220995 18854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:43.221259 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:43.221323 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714572529860089-16439-0/minicluster-data/ts-2-root
uuid: "a67cb959fb7349c09d3b52e121b5e8e7"
format_stamp: "Formatted at 2024-05-01 14:09:43 on dist-test-slave-mgxc"
I20240501 14:09:43.221392 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714572529860089-16439-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714572529860089-16439-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714572529860089-16439-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:43.225905 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:43.226243 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:43.226516 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:43.226922 16439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:43.226956 16439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:43.226989 16439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:43.227022 16439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:43.231303 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.195:43095
I20240501 14:09:43.231381 18917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.195:43095 every 8 connection(s)
I20240501 14:09:43.235594 18918 heartbeater.cc:347] Connected to a master server at 127.16.13.254:38013
I20240501 14:09:43.235714 18918 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:43.235978 18918 heartbeater.cc:511] Master 127.16.13.254:38013 requested a full tablet report, sending...
I20240501 14:09:43.236645 18638 ts_manager.cc:194] Registered new tserver with Master: a67cb959fb7349c09d3b52e121b5e8e7 (127.16.13.195:43095)
I20240501 14:09:43.237125 18638 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59556
I20240501 14:09:43.237351 16439 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005660979s
I20240501 14:09:44.178614 18767 heartbeater.cc:503] Master 127.16.13.254:38013 was elected leader, sending a full tablet report...
I20240501 14:09:44.218288 18843 heartbeater.cc:503] Master 127.16.13.254:38013 was elected leader, sending a full tablet report...
I20240501 14:09:44.238126 18918 heartbeater.cc:503] Master 127.16.13.254:38013 was elected leader, sending a full tablet report...
I20240501 14:09:45.108143 18692 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:09:47.108522 18692 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:09:49.108870 18692 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:09:51.109231 18692 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:09:53.109607 18692 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:09:55.110042 18692 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:09:55.316154 16439 tablet_server.cc:178] TabletServer@127.16.13.193:0 shutting down...
I20240501 14:09:55.320226 16439 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:09:55.332267 16439 tablet_server.cc:194] TabletServer@127.16.13.193:0 shutdown complete.
I20240501 14:09:55.334121 16439 tablet_server.cc:178] TabletServer@127.16.13.194:0 shutting down...
I20240501 14:09:55.337631 16439 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:09:55.349382 16439 tablet_server.cc:194] TabletServer@127.16.13.194:0 shutdown complete.
I20240501 14:09:55.351119 16439 tablet_server.cc:178] TabletServer@127.16.13.195:0 shutting down...
I20240501 14:09:55.354331 16439 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:09:55.366034 16439 tablet_server.cc:194] TabletServer@127.16.13.195:0 shutdown complete.
I20240501 14:09:55.367676 16439 master.cc:537] Master@127.16.13.254:38013 shutting down...
I20240501 14:09:55.370472 16439 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 519862f6785d42ec834085dbbcef9c75 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:55.370616 16439 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 519862f6785d42ec834085dbbcef9c75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:55.370666 16439 tablet_replica.cc:330] T 00000000000000000000000000000000 P 519862f6785d42ec834085dbbcef9c75: stopping tablet replica
I20240501 14:09:55.382848 16439 master.cc:552] Master@127.16.13.254:38013 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12350 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20240501 14:09:55.387966 16439 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.13.254:32859
I20240501 14:09:55.388264 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 14:09:55.391211 16439 server_base.cc:1034] running on GCE node
W20240501 14:09:55.391875 18922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:55.392503 18923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:55.393208 18925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:55.393401 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:09:55.393472 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:09:55.393488 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572595393488 us; error 0 us; skew 500 ppm
I20240501 14:09:55.398746 16439 webserver.cc:460] Webserver started at http://127.16.13.254:32843/ using document root <none> and password file <none>
I20240501 14:09:55.398929 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:55.398999 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:55.399117 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:55.399637 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714572529860089-16439-0/minicluster-data/master-0-root/instance:
uuid: "1f3902c49e0a4f87b9ddf6051141579a"
format_stamp: "Formatted at 2024-05-01 14:09:55 on dist-test-slave-mgxc"
I20240501 14:09:55.401283 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:55.402252 18930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:55.402501 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:55.402575 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714572529860089-16439-0/minicluster-data/master-0-root
uuid: "1f3902c49e0a4f87b9ddf6051141579a"
format_stamp: "Formatted at 2024-05-01 14:09:55 on dist-test-slave-mgxc"
I20240501 14:09:55.402649 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714572529860089-16439-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714572529860089-16439-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714572529860089-16439-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:55.417040 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:55.417428 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:55.421277 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.254:32859
I20240501 14:09:55.422611 18982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.254:32859 every 8 connection(s)
I20240501 14:09:55.423033 18983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:55.424865 18983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f3902c49e0a4f87b9ddf6051141579a: Bootstrap starting.
I20240501 14:09:55.425606 18983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f3902c49e0a4f87b9ddf6051141579a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.426554 18983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f3902c49e0a4f87b9ddf6051141579a: No bootstrap required, opened a new log
I20240501 14:09:55.426894 18983 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f3902c49e0a4f87b9ddf6051141579a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3902c49e0a4f87b9ddf6051141579a" member_type: VOTER }
I20240501 14:09:55.426980 18983 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f3902c49e0a4f87b9ddf6051141579a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.427014 18983 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f3902c49e0a4f87b9ddf6051141579a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f3902c49e0a4f87b9ddf6051141579a, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.427152 18983 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f3902c49e0a4f87b9ddf6051141579a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3902c49e0a4f87b9ddf6051141579a" member_type: VOTER }
I20240501 14:09:55.427234 18983 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f3902c49e0a4f87b9ddf6051141579a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:09:55.427278 18983 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f3902c49e0a4f87b9ddf6051141579a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:09:55.427330 18983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f3902c49e0a4f87b9ddf6051141579a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.428001 18983 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f3902c49e0a4f87b9ddf6051141579a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3902c49e0a4f87b9ddf6051141579a" member_type: VOTER }
I20240501 14:09:55.428119 18983 leader_election.cc:302] T 00000000000000000000000000000000 P 1f3902c49e0a4f87b9ddf6051141579a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f3902c49e0a4f87b9ddf6051141579a; no voters:
I20240501 14:09:55.428287 18983 leader_election.cc:288] T 00000000000000000000000000000000 P 1f3902c49e0a4f87b9ddf6051141579a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:09:55.428377 18986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f3902c49e0a4f87b9ddf6051141579a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:55.428583 18986 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f3902c49e0a4f87b9ddf6051141579a [term 1 LEADER]: Becoming Leader. State: Replica: 1f3902c49e0a4f87b9ddf6051141579a, State: Running, Role: LEADER
I20240501 14:09:55.428710 18986 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f3902c49e0a4f87b9ddf6051141579a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3902c49e0a4f87b9ddf6051141579a" member_type: VOTER }
I20240501 14:09:55.428733 18983 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f3902c49e0a4f87b9ddf6051141579a [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:09:55.429102 18988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f3902c49e0a4f87b9ddf6051141579a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f3902c49e0a4f87b9ddf6051141579a. Latest consensus state: current_term: 1 leader_uuid: "1f3902c49e0a4f87b9ddf6051141579a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3902c49e0a4f87b9ddf6051141579a" member_type: VOTER } }
I20240501 14:09:55.429086 18987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f3902c49e0a4f87b9ddf6051141579a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f3902c49e0a4f87b9ddf6051141579a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3902c49e0a4f87b9ddf6051141579a" member_type: VOTER } }
I20240501 14:09:55.429193 18988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f3902c49e0a4f87b9ddf6051141579a [sys.catalog]: This master's current role is: LEADER
I20240501 14:09:55.429205 18987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f3902c49e0a4f87b9ddf6051141579a [sys.catalog]: This master's current role is: LEADER
I20240501 14:09:55.429733 18990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:09:55.430706 18990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:09:55.430924 16439 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 14:09:55.432425 18990 catalog_manager.cc:1347] Generated new cluster ID: 8346c014a23548b397cbeba2778f819e
I20240501 14:09:55.432483 18990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:09:55.445982 18990 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:09:55.446552 18990 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:09:55.457825 18990 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1f3902c49e0a4f87b9ddf6051141579a: Generated new TSK 0
I20240501 14:09:55.458010 18990 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:09:55.463042 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:55.465463 19004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:55.466218 19005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:55.466913 19007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:55.467834 16439 server_base.cc:1034] running on GCE node
I20240501 14:09:55.468037 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:09:55.468083 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:09:55.468101 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572595468099 us; error 0 us; skew 500 ppm
I20240501 14:09:55.468956 16439 webserver.cc:460] Webserver started at http://127.16.13.193:43457/ using document root <none> and password file <none>
I20240501 14:09:55.469111 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:55.469154 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:55.469228 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:55.469586 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714572529860089-16439-0/minicluster-data/ts-0-root/instance:
uuid: "e7a0d8ad332e49ee833bd4d683a67776"
format_stamp: "Formatted at 2024-05-01 14:09:55 on dist-test-slave-mgxc"
I20240501 14:09:55.470952 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:55.471817 19012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:55.472052 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:55.472132 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714572529860089-16439-0/minicluster-data/ts-0-root
uuid: "e7a0d8ad332e49ee833bd4d683a67776"
format_stamp: "Formatted at 2024-05-01 14:09:55 on dist-test-slave-mgxc"
I20240501 14:09:55.472221 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714572529860089-16439-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714572529860089-16439-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714572529860089-16439-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:55.497309 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:55.497706 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:55.497997 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:55.498482 16439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:55.498524 16439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:55.498570 16439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:55.498605 16439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:55.502573 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.193:34091
I20240501 14:09:55.502628 19075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.193:34091 every 8 connection(s)
I20240501 14:09:55.503353 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:55.506064 19081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:55.506604 19080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:55.507470 19083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:55.507787 19076 heartbeater.cc:347] Connected to a master server at 127.16.13.254:32859
I20240501 14:09:55.507869 19076 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:55.508101 19076 heartbeater.cc:511] Master 127.16.13.254:32859 requested a full tablet report, sending...
I20240501 14:09:55.508338 16439 server_base.cc:1034] running on GCE node
I20240501 14:09:55.508487 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:09:55.508524 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:09:55.508548 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572595508546 us; error 0 us; skew 500 ppm
I20240501 14:09:55.508752 18947 ts_manager.cc:194] Registered new tserver with Master: e7a0d8ad332e49ee833bd4d683a67776 (127.16.13.193:34091)
I20240501 14:09:55.509460 16439 webserver.cc:460] Webserver started at http://127.16.13.194:38989/ using document root <none> and password file <none>
I20240501 14:09:55.509538 18947 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44060
I20240501 14:09:55.509604 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:55.509660 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:55.509752 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:55.510164 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714572529860089-16439-0/minicluster-data/ts-1-root/instance:
uuid: "a91400ee53134046a985e8c77c024c11"
format_stamp: "Formatted at 2024-05-01 14:09:55 on dist-test-slave-mgxc"
I20240501 14:09:55.511587 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:55.512574 19088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:55.512835 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:55.512902 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714572529860089-16439-0/minicluster-data/ts-1-root
uuid: "a91400ee53134046a985e8c77c024c11"
format_stamp: "Formatted at 2024-05-01 14:09:55 on dist-test-slave-mgxc"
I20240501 14:09:55.512967 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714572529860089-16439-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714572529860089-16439-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714572529860089-16439-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:55.527575 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:55.527966 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:55.528247 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:55.528713 16439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:55.528769 16439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:55.528810 16439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:55.528825 16439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:55.533084 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.194:36045
I20240501 14:09:55.533154 19151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.194:36045 every 8 connection(s)
I20240501 14:09:55.533798 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:55.536456 19155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:55.537300 19156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:55.537904 19152 heartbeater.cc:347] Connected to a master server at 127.16.13.254:32859
I20240501 14:09:55.538007 19152 heartbeater.cc:464] Registering TS with master...
W20240501 14:09:55.538017 19158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:55.538277 19152 heartbeater.cc:511] Master 127.16.13.254:32859 requested a full tablet report, sending...
I20240501 14:09:55.538749 16439 server_base.cc:1034] running on GCE node
I20240501 14:09:55.538864 18947 ts_manager.cc:194] Registered new tserver with Master: a91400ee53134046a985e8c77c024c11 (127.16.13.194:36045)
I20240501 14:09:55.538939 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:09:55.538987 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:09:55.539016 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572595539016 us; error 0 us; skew 500 ppm
I20240501 14:09:55.539462 18947 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44070
I20240501 14:09:55.540093 16439 webserver.cc:460] Webserver started at http://127.16.13.195:42889/ using document root <none> and password file <none>
I20240501 14:09:55.540249 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:55.540300 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:55.540374 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:55.540843 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714572529860089-16439-0/minicluster-data/ts-2-root/instance:
uuid: "4b58cc73615b48bf958b430145de2138"
format_stamp: "Formatted at 2024-05-01 14:09:55 on dist-test-slave-mgxc"
I20240501 14:09:55.542435 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 14:09:55.543345 19163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:55.543560 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:55.543630 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714572529860089-16439-0/minicluster-data/ts-2-root
uuid: "4b58cc73615b48bf958b430145de2138"
format_stamp: "Formatted at 2024-05-01 14:09:55 on dist-test-slave-mgxc"
I20240501 14:09:55.543702 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714572529860089-16439-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714572529860089-16439-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714572529860089-16439-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:55.552989 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:55.553362 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:55.553687 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:55.554143 16439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:55.554188 16439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:55.554239 16439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:55.554275 16439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:55.559271 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.195:43661
I20240501 14:09:55.559310 19226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.195:43661 every 8 connection(s)
I20240501 14:09:55.560075 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:55.562403 19230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:55.563081 19231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:55.564013 19233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:55.564249 19227 heartbeater.cc:347] Connected to a master server at 127.16.13.254:32859
I20240501 14:09:55.564359 19227 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:55.564576 19227 heartbeater.cc:511] Master 127.16.13.254:32859 requested a full tablet report, sending...
I20240501 14:09:55.564901 16439 server_base.cc:1034] running on GCE node
I20240501 14:09:55.565163 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:09:55.565223 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:09:55.565248 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572595565248 us; error 0 us; skew 500 ppm
I20240501 14:09:55.565284 18947 ts_manager.cc:194] Registered new tserver with Master: 4b58cc73615b48bf958b430145de2138 (127.16.13.195:43661)
I20240501 14:09:55.565941 18947 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44084
I20240501 14:09:55.566246 16439 webserver.cc:460] Webserver started at http://127.16.13.196:42359/ using document root <none> and password file <none>
I20240501 14:09:55.566418 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:55.566478 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:55.566578 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:55.567008 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714572529860089-16439-0/minicluster-data/ts-3-root/instance:
uuid: "6b619f16dc384f2ea3bf7b0dd7d35b11"
format_stamp: "Formatted at 2024-05-01 14:09:55 on dist-test-slave-mgxc"
I20240501 14:09:55.568589 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:55.569525 19238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:55.569751 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:55.569823 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714572529860089-16439-0/minicluster-data/ts-3-root
uuid: "6b619f16dc384f2ea3bf7b0dd7d35b11"
format_stamp: "Formatted at 2024-05-01 14:09:55 on dist-test-slave-mgxc"
I20240501 14:09:55.569896 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714572529860089-16439-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714572529860089-16439-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714572529860089-16439-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:55.579805 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:55.580231 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:55.580541 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:55.580984 16439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:55.581028 16439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:55.581079 16439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:55.581115 16439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:55.585711 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.196:37943
I20240501 14:09:55.585757 19301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.196:37943 every 8 connection(s)
I20240501 14:09:55.590051 19302 heartbeater.cc:347] Connected to a master server at 127.16.13.254:32859
I20240501 14:09:55.590163 19302 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:55.590346 19302 heartbeater.cc:511] Master 127.16.13.254:32859 requested a full tablet report, sending...
I20240501 14:09:55.591025 18947 ts_manager.cc:194] Registered new tserver with Master: 6b619f16dc384f2ea3bf7b0dd7d35b11 (127.16.13.196:37943)
I20240501 14:09:55.591526 18947 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44096
I20240501 14:09:55.591720 16439 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.005644019s
I20240501 14:09:56.510596 19076 heartbeater.cc:503] Master 127.16.13.254:32859 was elected leader, sending a full tablet report...
I20240501 14:09:56.540508 19152 heartbeater.cc:503] Master 127.16.13.254:32859 was elected leader, sending a full tablet report...
I20240501 14:09:56.566908 19227 heartbeater.cc:503] Master 127.16.13.254:32859 was elected leader, sending a full tablet report...
I20240501 14:09:56.592483 19302 heartbeater.cc:503] Master 127.16.13.254:32859 was elected leader, sending a full tablet report...
I20240501 14:09:56.622459 16439 test_util.cc:268] Using random seed: 177222777
I20240501 14:09:56.629982 18947 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44112:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:09:56.638890 19191 tablet_service.cc:1467] Processing CreateTablet for tablet e9298bb019f74a54a53791c4fb137287 (DEFAULT_TABLE table=test-workload [id=0d6b8947939b487bb02edf3bdb2cc725]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:56.639245 19191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9298bb019f74a54a53791c4fb137287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.640166 19116 tablet_service.cc:1467] Processing CreateTablet for tablet e9298bb019f74a54a53791c4fb137287 (DEFAULT_TABLE table=test-workload [id=0d6b8947939b487bb02edf3bdb2cc725]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:56.640408 19116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9298bb019f74a54a53791c4fb137287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.641697 19323 tablet_bootstrap.cc:492] T e9298bb019f74a54a53791c4fb137287 P 4b58cc73615b48bf958b430145de2138: Bootstrap starting.
I20240501 14:09:56.641930 19266 tablet_service.cc:1467] Processing CreateTablet for tablet e9298bb019f74a54a53791c4fb137287 (DEFAULT_TABLE table=test-workload [id=0d6b8947939b487bb02edf3bdb2cc725]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:56.642177 19266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9298bb019f74a54a53791c4fb137287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.642712 19323 tablet_bootstrap.cc:654] T e9298bb019f74a54a53791c4fb137287 P 4b58cc73615b48bf958b430145de2138: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.643757 19323 tablet_bootstrap.cc:492] T e9298bb019f74a54a53791c4fb137287 P 4b58cc73615b48bf958b430145de2138: No bootstrap required, opened a new log
I20240501 14:09:56.643859 19323 ts_tablet_manager.cc:1367] T e9298bb019f74a54a53791c4fb137287 P 4b58cc73615b48bf958b430145de2138: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:56.644279 19324 tablet_bootstrap.cc:492] T e9298bb019f74a54a53791c4fb137287 P a91400ee53134046a985e8c77c024c11: Bootstrap starting.
I20240501 14:09:56.644337 19326 tablet_bootstrap.cc:492] T e9298bb019f74a54a53791c4fb137287 P 6b619f16dc384f2ea3bf7b0dd7d35b11: Bootstrap starting.
I20240501 14:09:56.644603 19323 raft_consensus.cc:348] T e9298bb019f74a54a53791c4fb137287 P 4b58cc73615b48bf958b430145de2138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b58cc73615b48bf958b430145de2138" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43661 } } peers { permanent_uuid: "a91400ee53134046a985e8c77c024c11" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 36045 } } peers { permanent_uuid: "6b619f16dc384f2ea3bf7b0dd7d35b11" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 37943 } }
I20240501 14:09:56.644755 19323 raft_consensus.cc:374] T e9298bb019f74a54a53791c4fb137287 P 4b58cc73615b48bf958b430145de2138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.644831 19323 raft_consensus.cc:729] T e9298bb019f74a54a53791c4fb137287 P 4b58cc73615b48bf958b430145de2138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b58cc73615b48bf958b430145de2138, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.644989 19323 consensus_queue.cc:260] T e9298bb019f74a54a53791c4fb137287 P 4b58cc73615b48bf958b430145de2138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b58cc73615b48bf958b430145de2138" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43661 } } peers { permanent_uuid: "a91400ee53134046a985e8c77c024c11" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 36045 } } peers { permanent_uuid: "6b619f16dc384f2ea3bf7b0dd7d35b11" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 37943 } }
I20240501 14:09:56.645278 19326 tablet_bootstrap.cc:654] T e9298bb019f74a54a53791c4fb137287 P 6b619f16dc384f2ea3bf7b0dd7d35b11: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.645377 19324 tablet_bootstrap.cc:654] T e9298bb019f74a54a53791c4fb137287 P a91400ee53134046a985e8c77c024c11: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.645426 19323 ts_tablet_manager.cc:1398] T e9298bb019f74a54a53791c4fb137287 P 4b58cc73615b48bf958b430145de2138: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:56.646441 19326 tablet_bootstrap.cc:492] T e9298bb019f74a54a53791c4fb137287 P 6b619f16dc384f2ea3bf7b0dd7d35b11: No bootstrap required, opened a new log
I20240501 14:09:56.646441 19324 tablet_bootstrap.cc:492] T e9298bb019f74a54a53791c4fb137287 P a91400ee53134046a985e8c77c024c11: No bootstrap required, opened a new log
I20240501 14:09:56.646590 19326 ts_tablet_manager.cc:1367] T e9298bb019f74a54a53791c4fb137287 P 6b619f16dc384f2ea3bf7b0dd7d35b11: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:56.646598 19324 ts_tablet_manager.cc:1367] T e9298bb019f74a54a53791c4fb137287 P a91400ee53134046a985e8c77c024c11: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:56.647058 19324 raft_consensus.cc:348] T e9298bb019f74a54a53791c4fb137287 P a91400ee53134046a985e8c77c024c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b58cc73615b48bf958b430145de2138" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43661 } } peers { permanent_uuid: "a91400ee53134046a985e8c77c024c11" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 36045 } } peers { permanent_uuid: "6b619f16dc384f2ea3bf7b0dd7d35b11" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 37943 } }
I20240501 14:09:56.647058 19326 raft_consensus.cc:348] T e9298bb019f74a54a53791c4fb137287 P 6b619f16dc384f2ea3bf7b0dd7d35b11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b58cc73615b48bf958b430145de2138" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43661 } } peers { permanent_uuid: "a91400ee53134046a985e8c77c024c11" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 36045 } } peers { permanent_uuid: "6b619f16dc384f2ea3bf7b0dd7d35b11" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 37943 } }
I20240501 14:09:56.647210 19324 raft_consensus.cc:374] T e9298bb019f74a54a53791c4fb137287 P a91400ee53134046a985e8c77c024c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.647228 19326 raft_consensus.cc:374] T e9298bb019f74a54a53791c4fb137287 P 6b619f16dc384f2ea3bf7b0dd7d35b11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.647259 19324 raft_consensus.cc:729] T e9298bb019f74a54a53791c4fb137287 P a91400ee53134046a985e8c77c024c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91400ee53134046a985e8c77c024c11, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.647266 19326 raft_consensus.cc:729] T e9298bb019f74a54a53791c4fb137287 P 6b619f16dc384f2ea3bf7b0dd7d35b11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b619f16dc384f2ea3bf7b0dd7d35b11, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.647419 19324 consensus_queue.cc:260] T e9298bb019f74a54a53791c4fb137287 P a91400ee53134046a985e8c77c024c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b58cc73615b48bf958b430145de2138" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43661 } } peers { permanent_uuid: "a91400ee53134046a985e8c77c024c11" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 36045 } } peers { permanent_uuid: "6b619f16dc384f2ea3bf7b0dd7d35b11" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 37943 } }
I20240501 14:09:56.647414 19326 consensus_queue.cc:260] T e9298bb019f74a54a53791c4fb137287 P 6b619f16dc384f2ea3bf7b0dd7d35b11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b58cc73615b48bf958b430145de2138" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43661 } } peers { permanent_uuid: "a91400ee53134046a985e8c77c024c11" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 36045 } } peers { permanent_uuid: "6b619f16dc384f2ea3bf7b0dd7d35b11" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 37943 } }
I20240501 14:09:56.647779 19326 ts_tablet_manager.cc:1398] T e9298bb019f74a54a53791c4fb137287 P 6b619f16dc384f2ea3bf7b0dd7d35b11: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:56.647779 19324 ts_tablet_manager.cc:1398] T e9298bb019f74a54a53791c4fb137287 P a91400ee53134046a985e8c77c024c11: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240501 14:09:56.784482 19153 tablet.cc:2369] T e9298bb019f74a54a53791c4fb137287 P a91400ee53134046a985e8c77c024c11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:56.835666 19327 raft_consensus.cc:482] T e9298bb019f74a54a53791c4fb137287 P 4b58cc73615b48bf958b430145de2138 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:56.835880 19327 raft_consensus.cc:504] T e9298bb019f74a54a53791c4fb137287 P 4b58cc73615b48bf958b430145de2138 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b58cc73615b48bf958b430145de2138" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43661 } } peers { permanent_uuid: "a91400ee53134046a985e8c77c024c11" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 36045 } } peers { permanent_uuid: "6b619f16dc384f2ea3bf7b0dd7d35b11" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 37943 } }
I20240501 14:09:56.836633 19327 leader_election.cc:288] T e9298bb019f74a54a53791c4fb137287 P 4b58cc73615b48bf958b430145de2138 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91400ee53134046a985e8c77c024c11 (127.16.13.194:36045), 6b619f16dc384f2ea3bf7b0dd7d35b11 (127.16.13.196:37943)
I20240501 14:09:56.840083 19276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9298bb019f74a54a53791c4fb137287" candidate_uuid: "4b58cc73615b48bf958b430145de2138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b619f16dc384f2ea3bf7b0dd7d35b11" is_pre_election: true
I20240501 14:09:56.840231 19276 raft_consensus.cc:2454] T e9298bb019f74a54a53791c4fb137287 P 6b619f16dc384f2ea3bf7b0dd7d35b11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b58cc73615b48bf958b430145de2138 in term 0.
I20240501 14:09:56.840237 19126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9298bb019f74a54a53791c4fb137287" candidate_uuid: "4b58cc73615b48bf958b430145de2138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91400ee53134046a985e8c77c024c11" is_pre_election: true
I20240501 14:09:56.840441 19126 raft_consensus.cc:2454] T e9298bb019f74a54a53791c4fb137287 P a91400ee53134046a985e8c77c024c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b58cc73615b48bf958b430145de2138 in term 0.
I20240501 14:09:56.840555 19166 leader_election.cc:302] T e9298bb019f74a54a53791c4fb137287 P 4b58cc73615b48bf958b430145de2138 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b58cc73615b48bf958b430145de2138, 6b619f16dc384f2ea3bf7b0dd7d35b11; no voters:
I20240501 14:09:56.840732 19327 raft_consensus.cc:2781] T e9298bb019f74a54a53791c4fb137287 P 4b58cc73615b48bf958b430145de2138 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:56.840785 19327 raft_consensus.cc:482] T e9298bb019f74a54a53791c4fb137287 P 4b58cc73615b48bf958b430145de2138 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:56.840821 19327 raft_consensus.cc:3037] T e9298bb019f74a54a53791c4fb137287 P 4b58cc73615b48bf958b430145de2138 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:56.841637 19327 raft_consensus.cc:504] T e9298bb019f74a54a53791c4fb137287 P 4b58cc73615b48bf958b430145de2138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b58cc73615b48bf958b430145de2138" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43661 } } peers { permanent_uuid: "a91400ee53134046a985e8c77c024c11" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 36045 } } peers { permanent_uuid: "6b619f16dc384f2ea3bf7b0dd7d35b11" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 37943 } }
I20240501 14:09:56.841943 19327 leader_election.cc:288] T e9298bb019f74a54a53791c4fb137287 P 4b58cc73615b48bf958b430145de2138 [CANDIDATE]: Term 1 election: Requested vote from peers a91400ee53134046a985e8c77c024c11 (127.16.13.194:36045), 6b619f16dc384f2ea3bf7b0dd7d35b11 (127.16.13.196:37943)
I20240501 14:09:56.842232 19276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9298bb019f74a54a53791c4fb137287" candidate_uuid: "4b58cc73615b48bf958b430145de2138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b619f16dc384f2ea3bf7b0dd7d35b11"
I20240501 14:09:56.842242 19126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9298bb019f74a54a53791c4fb137287" candidate_uuid: "4b58cc73615b48bf958b430145de2138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91400ee53134046a985e8c77c024c11"
I20240501 14:09:56.842341 19126 raft_consensus.cc:3037] T e9298bb019f74a54a53791c4fb137287 P a91400ee53134046a985e8c77c024c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:56.842348 19276 raft_consensus.cc:3037] T e9298bb019f74a54a53791c4fb137287 P 6b619f16dc384f2ea3bf7b0dd7d35b11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:56.843191 19126 raft_consensus.cc:2454] T e9298bb019f74a54a53791c4fb137287 P a91400ee53134046a985e8c77c024c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b58cc73615b48bf958b430145de2138 in term 1.
I20240501 14:09:56.843192 19276 raft_consensus.cc:2454] T e9298bb019f74a54a53791c4fb137287 P 6b619f16dc384f2ea3bf7b0dd7d35b11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b58cc73615b48bf958b430145de2138 in term 1.
I20240501 14:09:56.843461 19167 leader_election.cc:302] T e9298bb019f74a54a53791c4fb137287 P 4b58cc73615b48bf958b430145de2138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b58cc73615b48bf958b430145de2138, a91400ee53134046a985e8c77c024c11; no voters:
I20240501 14:09:56.843655 19327 raft_consensus.cc:2781] T e9298bb019f74a54a53791c4fb137287 P 4b58cc73615b48bf958b430145de2138 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:56.843840 19327 raft_consensus.cc:686] T e9298bb019f74a54a53791c4fb137287 P 4b58cc73615b48bf958b430145de2138 [term 1 LEADER]: Becoming Leader. State: Replica: 4b58cc73615b48bf958b430145de2138, State: Running, Role: LEADER
I20240501 14:09:56.843988 19327 consensus_queue.cc:237] T e9298bb019f74a54a53791c4fb137287 P 4b58cc73615b48bf958b430145de2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b58cc73615b48bf958b430145de2138" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43661 } } peers { permanent_uuid: "a91400ee53134046a985e8c77c024c11" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 36045 } } peers { permanent_uuid: "6b619f16dc384f2ea3bf7b0dd7d35b11" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 37943 } }
I20240501 14:09:56.845387 18946 catalog_manager.cc:5484] T e9298bb019f74a54a53791c4fb137287 P 4b58cc73615b48bf958b430145de2138 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b58cc73615b48bf958b430145de2138 (127.16.13.195). New cstate: current_term: 1 leader_uuid: "4b58cc73615b48bf958b430145de2138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b58cc73615b48bf958b430145de2138" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 43661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a91400ee53134046a985e8c77c024c11" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 36045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b619f16dc384f2ea3bf7b0dd7d35b11" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 37943 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:57.221702 19327 consensus_queue.cc:1035] T e9298bb019f74a54a53791c4fb137287 P 4b58cc73615b48bf958b430145de2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b619f16dc384f2ea3bf7b0dd7d35b11" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 37943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:57.224835 19339 consensus_queue.cc:1035] T e9298bb019f74a54a53791c4fb137287 P 4b58cc73615b48bf958b430145de2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91400ee53134046a985e8c77c024c11" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 36045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:57.430795 19001 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:09:57.431102 19001 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:09:57.431155 19001 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:09:59.431458 19001 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:09:59.431748 19001 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:09:59.431800 19001 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:10:01.432126 19001 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:10:01.432439 19001 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:10:01.432493 19001 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:10:03.432816 19001 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:10:03.433159 19001 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:10:03.433213 19001 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:10:05.433542 19001 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:10:05.433951 19001 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:10:05.434087 19001 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:10:07.434437 19001 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:10:07.434734 19001 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:10:07.434787 19001 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:10:07.919382 16439 tablet_server.cc:178] TabletServer@127.16.13.193:0 shutting down...
I20240501 14:10:07.922966 16439 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:10:07.934911 16439 tablet_server.cc:194] TabletServer@127.16.13.193:0 shutdown complete.
I20240501 14:10:07.936829 16439 tablet_server.cc:178] TabletServer@127.16.13.194:0 shutting down...
I20240501 14:10:07.940227 16439 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:10:07.940407 16439 tablet_replica.cc:330] T e9298bb019f74a54a53791c4fb137287 P a91400ee53134046a985e8c77c024c11: stopping tablet replica
I20240501 14:10:07.940528 16439 raft_consensus.cc:2229] T e9298bb019f74a54a53791c4fb137287 P a91400ee53134046a985e8c77c024c11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:07.940627 16439 raft_consensus.cc:2258] T e9298bb019f74a54a53791c4fb137287 P a91400ee53134046a985e8c77c024c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:07.952870 16439 tablet_server.cc:194] TabletServer@127.16.13.194:0 shutdown complete.
I20240501 14:10:07.954712 16439 tablet_server.cc:178] TabletServer@127.16.13.195:0 shutting down...
I20240501 14:10:07.958186 16439 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:10:07.958343 16439 tablet_replica.cc:330] T e9298bb019f74a54a53791c4fb137287 P 4b58cc73615b48bf958b430145de2138: stopping tablet replica
I20240501 14:10:07.958448 16439 raft_consensus.cc:2229] T e9298bb019f74a54a53791c4fb137287 P 4b58cc73615b48bf958b430145de2138 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:10:07.958611 16439 raft_consensus.cc:2258] T e9298bb019f74a54a53791c4fb137287 P 4b58cc73615b48bf958b430145de2138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:07.971305 16439 tablet_server.cc:194] TabletServer@127.16.13.195:0 shutdown complete.
I20240501 14:10:07.973590 16439 tablet_server.cc:178] TabletServer@127.16.13.196:0 shutting down...
I20240501 14:10:07.976945 16439 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:10:07.977133 16439 tablet_replica.cc:330] T e9298bb019f74a54a53791c4fb137287 P 6b619f16dc384f2ea3bf7b0dd7d35b11: stopping tablet replica
I20240501 14:10:07.977248 16439 raft_consensus.cc:2229] T e9298bb019f74a54a53791c4fb137287 P 6b619f16dc384f2ea3bf7b0dd7d35b11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:07.977362 16439 raft_consensus.cc:2258] T e9298bb019f74a54a53791c4fb137287 P 6b619f16dc384f2ea3bf7b0dd7d35b11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:07.989603 16439 tablet_server.cc:194] TabletServer@127.16.13.196:0 shutdown complete.
I20240501 14:10:07.991562 16439 master.cc:537] Master@127.16.13.254:32859 shutting down...
I20240501 14:10:07.994694 16439 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1f3902c49e0a4f87b9ddf6051141579a [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:10:07.994846 16439 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1f3902c49e0a4f87b9ddf6051141579a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:07.994892 16439 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1f3902c49e0a4f87b9ddf6051141579a: stopping tablet replica
I20240501 14:10:08.007030 16439 master.cc:552] Master@127.16.13.254:32859 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12625 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20240501 14:10:08.013306 16439 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.13.254:39135
I20240501 14:10:08.013583 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:10:08.015722 19348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:08.016366 19351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:08.017185 19349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:10:08.018030 16439 server_base.cc:1034] running on GCE node
I20240501 14:10:08.018164 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:10:08.018200 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:10:08.018222 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572608018221 us; error 0 us; skew 500 ppm
I20240501 14:10:08.019109 16439 webserver.cc:460] Webserver started at http://127.16.13.254:43437/ using document root <none> and password file <none>
I20240501 14:10:08.019250 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:10:08.019300 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:10:08.019388 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:10:08.019752 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714572529860089-16439-0/minicluster-data/master-0-root/instance:
uuid: "b3d6369ed2fb496f8825d34ae65316f5"
format_stamp: "Formatted at 2024-05-01 14:10:08 on dist-test-slave-mgxc"
I20240501 14:10:08.021193 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:08.022071 19356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:08.022342 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:08.022423 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714572529860089-16439-0/minicluster-data/master-0-root
uuid: "b3d6369ed2fb496f8825d34ae65316f5"
format_stamp: "Formatted at 2024-05-01 14:10:08 on dist-test-slave-mgxc"
I20240501 14:10:08.022497 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714572529860089-16439-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714572529860089-16439-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714572529860089-16439-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:10:08.034400 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:10:08.034740 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:10:08.038858 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.254:39135
I20240501 14:10:08.040967 19408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.254:39135 every 8 connection(s)
I20240501 14:10:08.041428 19409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:08.043152 19409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3d6369ed2fb496f8825d34ae65316f5: Bootstrap starting.
I20240501 14:10:08.043898 19409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3d6369ed2fb496f8825d34ae65316f5: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:08.044854 19409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3d6369ed2fb496f8825d34ae65316f5: No bootstrap required, opened a new log
I20240501 14:10:08.045192 19409 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3d6369ed2fb496f8825d34ae65316f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3d6369ed2fb496f8825d34ae65316f5" member_type: VOTER }
I20240501 14:10:08.045275 19409 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3d6369ed2fb496f8825d34ae65316f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:08.045308 19409 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3d6369ed2fb496f8825d34ae65316f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3d6369ed2fb496f8825d34ae65316f5, State: Initialized, Role: FOLLOWER
I20240501 14:10:08.045445 19409 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3d6369ed2fb496f8825d34ae65316f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3d6369ed2fb496f8825d34ae65316f5" member_type: VOTER }
I20240501 14:10:08.045511 19409 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3d6369ed2fb496f8825d34ae65316f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:10:08.045553 19409 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3d6369ed2fb496f8825d34ae65316f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:10:08.045603 19409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3d6369ed2fb496f8825d34ae65316f5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:08.046198 19409 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3d6369ed2fb496f8825d34ae65316f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3d6369ed2fb496f8825d34ae65316f5" member_type: VOTER }
I20240501 14:10:08.046315 19409 leader_election.cc:302] T 00000000000000000000000000000000 P b3d6369ed2fb496f8825d34ae65316f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3d6369ed2fb496f8825d34ae65316f5; no voters:
I20240501 14:10:08.046478 19409 leader_election.cc:288] T 00000000000000000000000000000000 P b3d6369ed2fb496f8825d34ae65316f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:10:08.046567 19412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3d6369ed2fb496f8825d34ae65316f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:08.046749 19412 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3d6369ed2fb496f8825d34ae65316f5 [term 1 LEADER]: Becoming Leader. State: Replica: b3d6369ed2fb496f8825d34ae65316f5, State: Running, Role: LEADER
I20240501 14:10:08.046854 19409 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3d6369ed2fb496f8825d34ae65316f5 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:10:08.046893 19412 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3d6369ed2fb496f8825d34ae65316f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3d6369ed2fb496f8825d34ae65316f5" member_type: VOTER }
I20240501 14:10:08.047268 19413 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3d6369ed2fb496f8825d34ae65316f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3d6369ed2fb496f8825d34ae65316f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3d6369ed2fb496f8825d34ae65316f5" member_type: VOTER } }
I20240501 14:10:08.047312 19414 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3d6369ed2fb496f8825d34ae65316f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3d6369ed2fb496f8825d34ae65316f5. Latest consensus state: current_term: 1 leader_uuid: "b3d6369ed2fb496f8825d34ae65316f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3d6369ed2fb496f8825d34ae65316f5" member_type: VOTER } }
I20240501 14:10:08.047441 19414 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3d6369ed2fb496f8825d34ae65316f5 [sys.catalog]: This master's current role is: LEADER
I20240501 14:10:08.047425 19413 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3d6369ed2fb496f8825d34ae65316f5 [sys.catalog]: This master's current role is: LEADER
I20240501 14:10:08.047976 19418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:10:08.048861 19418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:10:08.049077 16439 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 14:10:08.050546 19418 catalog_manager.cc:1347] Generated new cluster ID: f251e6394d0c4ee798e7e73810ce862d
I20240501 14:10:08.050601 19418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:10:08.062888 19418 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:10:08.063380 19418 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:10:08.068816 19418 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b3d6369ed2fb496f8825d34ae65316f5: Generated new TSK 0
I20240501 14:10:08.068958 19418 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:10:08.081416 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:10:08.084332 19433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:08.085026 19430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:08.085644 19431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:10:08.086412 16439 server_base.cc:1034] running on GCE node
I20240501 14:10:08.086582 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:10:08.086623 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:10:08.086647 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572608086646 us; error 0 us; skew 500 ppm
I20240501 14:10:08.087565 16439 webserver.cc:460] Webserver started at http://127.16.13.193:33445/ using document root <none> and password file <none>
I20240501 14:10:08.087713 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:10:08.087767 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:10:08.087846 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:10:08.088295 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714572529860089-16439-0/minicluster-data/ts-0-root/instance:
uuid: "07c81dfeb3f74eef802de32f6f485512"
format_stamp: "Formatted at 2024-05-01 14:10:08 on dist-test-slave-mgxc"
I20240501 14:10:08.089797 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 14:10:08.090711 19438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:08.090926 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:08.090993 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714572529860089-16439-0/minicluster-data/ts-0-root
uuid: "07c81dfeb3f74eef802de32f6f485512"
format_stamp: "Formatted at 2024-05-01 14:10:08 on dist-test-slave-mgxc"
I20240501 14:10:08.091061 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714572529860089-16439-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714572529860089-16439-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714572529860089-16439-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:10:08.099131 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:10:08.099447 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:10:08.099699 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:10:08.100167 16439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:10:08.100207 16439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:08.100255 16439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:10:08.100287 16439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:08.104120 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.193:35541
I20240501 14:10:08.104164 19501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.193:35541 every 8 connection(s)
I20240501 14:10:08.104924 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:10:08.107496 19507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:08.108054 19506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:08.108709 19509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:10:08.109228 19502 heartbeater.cc:347] Connected to a master server at 127.16.13.254:39135
I20240501 14:10:08.109319 19502 heartbeater.cc:464] Registering TS with master...
I20240501 14:10:08.109647 16439 server_base.cc:1034] running on GCE node
I20240501 14:10:08.109676 19502 heartbeater.cc:511] Master 127.16.13.254:39135 requested a full tablet report, sending...
I20240501 14:10:08.109794 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:10:08.109839 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:10:08.109872 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572608109871 us; error 0 us; skew 500 ppm
I20240501 14:10:08.110363 19373 ts_manager.cc:194] Registered new tserver with Master: 07c81dfeb3f74eef802de32f6f485512 (127.16.13.193:35541)
I20240501 14:10:08.110833 16439 webserver.cc:460] Webserver started at http://127.16.13.194:34431/ using document root <none> and password file <none>
I20240501 14:10:08.110981 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:10:08.111025 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:10:08.111099 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:10:08.111179 19373 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54746
I20240501 14:10:08.111534 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714572529860089-16439-0/minicluster-data/ts-1-root/instance:
uuid: "619773ca0df3492e9d95c4052c203b41"
format_stamp: "Formatted at 2024-05-01 14:10:08 on dist-test-slave-mgxc"
I20240501 14:10:08.113013 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 14:10:08.113963 19514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:08.114234 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:08.114326 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714572529860089-16439-0/minicluster-data/ts-1-root
uuid: "619773ca0df3492e9d95c4052c203b41"
format_stamp: "Formatted at 2024-05-01 14:10:08 on dist-test-slave-mgxc"
I20240501 14:10:08.114390 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714572529860089-16439-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714572529860089-16439-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714572529860089-16439-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:10:08.128979 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:10:08.129336 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:10:08.129639 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:10:08.130060 16439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:10:08.130100 16439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:08.130148 16439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:10:08.130182 16439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:08.134204 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.194:40183
I20240501 14:10:08.134241 19577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.194:40183 every 8 connection(s)
I20240501 14:10:08.135018 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:10:08.137565 19581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:08.138414 19582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:10:08.139123 19578 heartbeater.cc:347] Connected to a master server at 127.16.13.254:39135
W20240501 14:10:08.139151 19584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:10:08.139212 19578 heartbeater.cc:464] Registering TS with master...
I20240501 14:10:08.139408 19578 heartbeater.cc:511] Master 127.16.13.254:39135 requested a full tablet report, sending...
I20240501 14:10:08.139817 16439 server_base.cc:1034] running on GCE node
I20240501 14:10:08.139981 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:10:08.140029 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:10:08.140050 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572608140050 us; error 0 us; skew 500 ppm
I20240501 14:10:08.140081 19373 ts_manager.cc:194] Registered new tserver with Master: 619773ca0df3492e9d95c4052c203b41 (127.16.13.194:40183)
I20240501 14:10:08.140699 19373 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54758
I20240501 14:10:08.141042 16439 webserver.cc:460] Webserver started at http://127.16.13.195:43975/ using document root <none> and password file <none>
I20240501 14:10:08.141193 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:10:08.141242 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:10:08.141467 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:10:08.141902 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714572529860089-16439-0/minicluster-data/ts-2-root/instance:
uuid: "4cde0e47c95341cbae7fde5be0d3aee7"
format_stamp: "Formatted at 2024-05-01 14:10:08 on dist-test-slave-mgxc"
I20240501 14:10:08.143247 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 14:10:08.144081 19589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:08.144349 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:08.144407 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714572529860089-16439-0/minicluster-data/ts-2-root
uuid: "4cde0e47c95341cbae7fde5be0d3aee7"
format_stamp: "Formatted at 2024-05-01 14:10:08 on dist-test-slave-mgxc"
I20240501 14:10:08.144457 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714572529860089-16439-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714572529860089-16439-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714572529860089-16439-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:10:08.160568 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:10:08.160919 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:10:08.161176 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:10:08.161648 16439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:10:08.161687 16439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:08.161721 16439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:10:08.161753 16439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:08.166129 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.195:46703
I20240501 14:10:08.166206 19652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.195:46703 every 8 connection(s)
I20240501 14:10:08.170333 19653 heartbeater.cc:347] Connected to a master server at 127.16.13.254:39135
I20240501 14:10:08.170424 19653 heartbeater.cc:464] Registering TS with master...
I20240501 14:10:08.170596 19653 heartbeater.cc:511] Master 127.16.13.254:39135 requested a full tablet report, sending...
I20240501 14:10:08.171133 19373 ts_manager.cc:194] Registered new tserver with Master: 4cde0e47c95341cbae7fde5be0d3aee7 (127.16.13.195:46703)
I20240501 14:10:08.171622 19373 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54768
I20240501 14:10:08.172108 16439 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005568764s
I20240501 14:10:09.112138 19502 heartbeater.cc:503] Master 127.16.13.254:39135 was elected leader, sending a full tablet report...
I20240501 14:10:09.141634 19578 heartbeater.cc:503] Master 127.16.13.254:39135 was elected leader, sending a full tablet report...
I20240501 14:10:09.172478 19653 heartbeater.cc:503] Master 127.16.13.254:39135 was elected leader, sending a full tablet report...
I20240501 14:10:09.202644 16439 test_util.cc:268] Using random seed: 189802961
I20240501 14:10:09.209815 19373 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54786:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 14:10:09.210289 19373 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 14:10:09.219642 19617 tablet_service.cc:1467] Processing CreateTablet for tablet 54dde18516574ee9a3c02dc730e926e7 (DEFAULT_TABLE table=test-workload [id=912a62bc05d14e8491799436ff33cead]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:09.219669 19466 tablet_service.cc:1467] Processing CreateTablet for tablet 54dde18516574ee9a3c02dc730e926e7 (DEFAULT_TABLE table=test-workload [id=912a62bc05d14e8491799436ff33cead]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:09.219794 19542 tablet_service.cc:1467] Processing CreateTablet for tablet 54dde18516574ee9a3c02dc730e926e7 (DEFAULT_TABLE table=test-workload [id=912a62bc05d14e8491799436ff33cead]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:09.220005 19466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54dde18516574ee9a3c02dc730e926e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:09.220005 19617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54dde18516574ee9a3c02dc730e926e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:09.220018 19542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54dde18516574ee9a3c02dc730e926e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:09.222157 19674 tablet_bootstrap.cc:492] T 54dde18516574ee9a3c02dc730e926e7 P 07c81dfeb3f74eef802de32f6f485512: Bootstrap starting.
I20240501 14:10:09.222566 19675 tablet_bootstrap.cc:492] T 54dde18516574ee9a3c02dc730e926e7 P 4cde0e47c95341cbae7fde5be0d3aee7: Bootstrap starting.
I20240501 14:10:09.223071 19674 tablet_bootstrap.cc:654] T 54dde18516574ee9a3c02dc730e926e7 P 07c81dfeb3f74eef802de32f6f485512: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:09.223436 19675 tablet_bootstrap.cc:654] T 54dde18516574ee9a3c02dc730e926e7 P 4cde0e47c95341cbae7fde5be0d3aee7: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:09.224197 19676 tablet_bootstrap.cc:492] T 54dde18516574ee9a3c02dc730e926e7 P 619773ca0df3492e9d95c4052c203b41: Bootstrap starting.
I20240501 14:10:09.224200 19674 tablet_bootstrap.cc:492] T 54dde18516574ee9a3c02dc730e926e7 P 07c81dfeb3f74eef802de32f6f485512: No bootstrap required, opened a new log
I20240501 14:10:09.224323 19674 ts_tablet_manager.cc:1367] T 54dde18516574ee9a3c02dc730e926e7 P 07c81dfeb3f74eef802de32f6f485512: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:09.224488 19675 tablet_bootstrap.cc:492] T 54dde18516574ee9a3c02dc730e926e7 P 4cde0e47c95341cbae7fde5be0d3aee7: No bootstrap required, opened a new log
I20240501 14:10:09.224563 19675 ts_tablet_manager.cc:1367] T 54dde18516574ee9a3c02dc730e926e7 P 4cde0e47c95341cbae7fde5be0d3aee7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:10:09.224766 19674 raft_consensus.cc:348] T 54dde18516574ee9a3c02dc730e926e7 P 07c81dfeb3f74eef802de32f6f485512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cde0e47c95341cbae7fde5be0d3aee7" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 46703 } } peers { permanent_uuid: "619773ca0df3492e9d95c4052c203b41" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 40183 } } peers { permanent_uuid: "07c81dfeb3f74eef802de32f6f485512" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 35541 } }
I20240501 14:10:09.224871 19674 raft_consensus.cc:374] T 54dde18516574ee9a3c02dc730e926e7 P 07c81dfeb3f74eef802de32f6f485512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:09.224946 19674 raft_consensus.cc:729] T 54dde18516574ee9a3c02dc730e926e7 P 07c81dfeb3f74eef802de32f6f485512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c81dfeb3f74eef802de32f6f485512, State: Initialized, Role: FOLLOWER
I20240501 14:10:09.224913 19675 raft_consensus.cc:348] T 54dde18516574ee9a3c02dc730e926e7 P 4cde0e47c95341cbae7fde5be0d3aee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cde0e47c95341cbae7fde5be0d3aee7" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 46703 } } peers { permanent_uuid: "619773ca0df3492e9d95c4052c203b41" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 40183 } } peers { permanent_uuid: "07c81dfeb3f74eef802de32f6f485512" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 35541 } }
I20240501 14:10:09.225016 19675 raft_consensus.cc:374] T 54dde18516574ee9a3c02dc730e926e7 P 4cde0e47c95341cbae7fde5be0d3aee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:09.225052 19676 tablet_bootstrap.cc:654] T 54dde18516574ee9a3c02dc730e926e7 P 619773ca0df3492e9d95c4052c203b41: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:09.225098 19675 raft_consensus.cc:729] T 54dde18516574ee9a3c02dc730e926e7 P 4cde0e47c95341cbae7fde5be0d3aee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cde0e47c95341cbae7fde5be0d3aee7, State: Initialized, Role: FOLLOWER
I20240501 14:10:09.225126 19674 consensus_queue.cc:260] T 54dde18516574ee9a3c02dc730e926e7 P 07c81dfeb3f74eef802de32f6f485512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cde0e47c95341cbae7fde5be0d3aee7" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 46703 } } peers { permanent_uuid: "619773ca0df3492e9d95c4052c203b41" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 40183 } } peers { permanent_uuid: "07c81dfeb3f74eef802de32f6f485512" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 35541 } }
I20240501 14:10:09.225198 19675 consensus_queue.cc:260] T 54dde18516574ee9a3c02dc730e926e7 P 4cde0e47c95341cbae7fde5be0d3aee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cde0e47c95341cbae7fde5be0d3aee7" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 46703 } } peers { permanent_uuid: "619773ca0df3492e9d95c4052c203b41" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 40183 } } peers { permanent_uuid: "07c81dfeb3f74eef802de32f6f485512" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 35541 } }
I20240501 14:10:09.225481 19675 ts_tablet_manager.cc:1398] T 54dde18516574ee9a3c02dc730e926e7 P 4cde0e47c95341cbae7fde5be0d3aee7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:09.225481 19674 ts_tablet_manager.cc:1398] T 54dde18516574ee9a3c02dc730e926e7 P 07c81dfeb3f74eef802de32f6f485512: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:09.226126 19676 tablet_bootstrap.cc:492] T 54dde18516574ee9a3c02dc730e926e7 P 619773ca0df3492e9d95c4052c203b41: No bootstrap required, opened a new log
I20240501 14:10:09.226219 19676 ts_tablet_manager.cc:1367] T 54dde18516574ee9a3c02dc730e926e7 P 619773ca0df3492e9d95c4052c203b41: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:09.226575 19676 raft_consensus.cc:348] T 54dde18516574ee9a3c02dc730e926e7 P 619773ca0df3492e9d95c4052c203b41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cde0e47c95341cbae7fde5be0d3aee7" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 46703 } } peers { permanent_uuid: "619773ca0df3492e9d95c4052c203b41" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 40183 } } peers { permanent_uuid: "07c81dfeb3f74eef802de32f6f485512" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 35541 } }
I20240501 14:10:09.226660 19676 raft_consensus.cc:374] T 54dde18516574ee9a3c02dc730e926e7 P 619773ca0df3492e9d95c4052c203b41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:09.226694 19676 raft_consensus.cc:729] T 54dde18516574ee9a3c02dc730e926e7 P 619773ca0df3492e9d95c4052c203b41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 619773ca0df3492e9d95c4052c203b41, State: Initialized, Role: FOLLOWER
I20240501 14:10:09.226982 19676 consensus_queue.cc:260] T 54dde18516574ee9a3c02dc730e926e7 P 619773ca0df3492e9d95c4052c203b41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cde0e47c95341cbae7fde5be0d3aee7" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 46703 } } peers { permanent_uuid: "619773ca0df3492e9d95c4052c203b41" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 40183 } } peers { permanent_uuid: "07c81dfeb3f74eef802de32f6f485512" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 35541 } }
I20240501 14:10:09.227365 19676 ts_tablet_manager.cc:1398] T 54dde18516574ee9a3c02dc730e926e7 P 619773ca0df3492e9d95c4052c203b41: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:09.502254 19679 raft_consensus.cc:482] T 54dde18516574ee9a3c02dc730e926e7 P 07c81dfeb3f74eef802de32f6f485512 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:09.502434 19679 raft_consensus.cc:504] T 54dde18516574ee9a3c02dc730e926e7 P 07c81dfeb3f74eef802de32f6f485512 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cde0e47c95341cbae7fde5be0d3aee7" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 46703 } } peers { permanent_uuid: "619773ca0df3492e9d95c4052c203b41" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 40183 } } peers { permanent_uuid: "07c81dfeb3f74eef802de32f6f485512" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 35541 } }
I20240501 14:10:09.503002 19679 leader_election.cc:288] T 54dde18516574ee9a3c02dc730e926e7 P 07c81dfeb3f74eef802de32f6f485512 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cde0e47c95341cbae7fde5be0d3aee7 (127.16.13.195:46703), 619773ca0df3492e9d95c4052c203b41 (127.16.13.194:40183)
I20240501 14:10:09.506709 19552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54dde18516574ee9a3c02dc730e926e7" candidate_uuid: "07c81dfeb3f74eef802de32f6f485512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "619773ca0df3492e9d95c4052c203b41" is_pre_election: true
I20240501 14:10:09.506711 19627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54dde18516574ee9a3c02dc730e926e7" candidate_uuid: "07c81dfeb3f74eef802de32f6f485512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cde0e47c95341cbae7fde5be0d3aee7" is_pre_election: true
I20240501 14:10:09.506848 19552 raft_consensus.cc:2454] T 54dde18516574ee9a3c02dc730e926e7 P 619773ca0df3492e9d95c4052c203b41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07c81dfeb3f74eef802de32f6f485512 in term 0.
I20240501 14:10:09.506877 19627 raft_consensus.cc:2454] T 54dde18516574ee9a3c02dc730e926e7 P 4cde0e47c95341cbae7fde5be0d3aee7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07c81dfeb3f74eef802de32f6f485512 in term 0.
I20240501 14:10:09.507171 19441 leader_election.cc:302] T 54dde18516574ee9a3c02dc730e926e7 P 07c81dfeb3f74eef802de32f6f485512 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c81dfeb3f74eef802de32f6f485512, 4cde0e47c95341cbae7fde5be0d3aee7; no voters:
I20240501 14:10:09.507366 19679 raft_consensus.cc:2781] T 54dde18516574ee9a3c02dc730e926e7 P 07c81dfeb3f74eef802de32f6f485512 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:09.507423 19679 raft_consensus.cc:482] T 54dde18516574ee9a3c02dc730e926e7 P 07c81dfeb3f74eef802de32f6f485512 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:09.507449 19679 raft_consensus.cc:3037] T 54dde18516574ee9a3c02dc730e926e7 P 07c81dfeb3f74eef802de32f6f485512 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:09.508344 19679 raft_consensus.cc:504] T 54dde18516574ee9a3c02dc730e926e7 P 07c81dfeb3f74eef802de32f6f485512 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cde0e47c95341cbae7fde5be0d3aee7" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 46703 } } peers { permanent_uuid: "619773ca0df3492e9d95c4052c203b41" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 40183 } } peers { permanent_uuid: "07c81dfeb3f74eef802de32f6f485512" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 35541 } }
I20240501 14:10:09.508677 19679 leader_election.cc:288] T 54dde18516574ee9a3c02dc730e926e7 P 07c81dfeb3f74eef802de32f6f485512 [CANDIDATE]: Term 1 election: Requested vote from peers 4cde0e47c95341cbae7fde5be0d3aee7 (127.16.13.195:46703), 619773ca0df3492e9d95c4052c203b41 (127.16.13.194:40183)
I20240501 14:10:09.509004 19627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54dde18516574ee9a3c02dc730e926e7" candidate_uuid: "07c81dfeb3f74eef802de32f6f485512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cde0e47c95341cbae7fde5be0d3aee7"
I20240501 14:10:09.508983 19552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54dde18516574ee9a3c02dc730e926e7" candidate_uuid: "07c81dfeb3f74eef802de32f6f485512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "619773ca0df3492e9d95c4052c203b41"
I20240501 14:10:09.509122 19627 raft_consensus.cc:3037] T 54dde18516574ee9a3c02dc730e926e7 P 4cde0e47c95341cbae7fde5be0d3aee7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:09.509135 19552 raft_consensus.cc:3037] T 54dde18516574ee9a3c02dc730e926e7 P 619773ca0df3492e9d95c4052c203b41 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:09.510172 19552 raft_consensus.cc:2454] T 54dde18516574ee9a3c02dc730e926e7 P 619773ca0df3492e9d95c4052c203b41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07c81dfeb3f74eef802de32f6f485512 in term 1.
I20240501 14:10:09.510262 19627 raft_consensus.cc:2454] T 54dde18516574ee9a3c02dc730e926e7 P 4cde0e47c95341cbae7fde5be0d3aee7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07c81dfeb3f74eef802de32f6f485512 in term 1.
I20240501 14:10:09.510506 19439 leader_election.cc:302] T 54dde18516574ee9a3c02dc730e926e7 P 07c81dfeb3f74eef802de32f6f485512 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c81dfeb3f74eef802de32f6f485512, 619773ca0df3492e9d95c4052c203b41; no voters:
I20240501 14:10:09.510696 19679 raft_consensus.cc:2781] T 54dde18516574ee9a3c02dc730e926e7 P 07c81dfeb3f74eef802de32f6f485512 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:09.510903 19679 raft_consensus.cc:686] T 54dde18516574ee9a3c02dc730e926e7 P 07c81dfeb3f74eef802de32f6f485512 [term 1 LEADER]: Becoming Leader. State: Replica: 07c81dfeb3f74eef802de32f6f485512, State: Running, Role: LEADER
I20240501 14:10:09.511056 19679 consensus_queue.cc:237] T 54dde18516574ee9a3c02dc730e926e7 P 07c81dfeb3f74eef802de32f6f485512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cde0e47c95341cbae7fde5be0d3aee7" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 46703 } } peers { permanent_uuid: "619773ca0df3492e9d95c4052c203b41" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 40183 } } peers { permanent_uuid: "07c81dfeb3f74eef802de32f6f485512" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 35541 } }
I20240501 14:10:09.512503 19370 catalog_manager.cc:5484] T 54dde18516574ee9a3c02dc730e926e7 P 07c81dfeb3f74eef802de32f6f485512 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07c81dfeb3f74eef802de32f6f485512 (127.16.13.193). New cstate: current_term: 1 leader_uuid: "07c81dfeb3f74eef802de32f6f485512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cde0e47c95341cbae7fde5be0d3aee7" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 46703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "619773ca0df3492e9d95c4052c203b41" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 40183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07c81dfeb3f74eef802de32f6f485512" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 35541 } health_report { overall_health: HEALTHY } } }
I20240501 14:10:09.896441 19679 consensus_queue.cc:1035] T 54dde18516574ee9a3c02dc730e926e7 P 07c81dfeb3f74eef802de32f6f485512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cde0e47c95341cbae7fde5be0d3aee7" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 46703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:09.899425 19679 consensus_queue.cc:1035] T 54dde18516574ee9a3c02dc730e926e7 P 07c81dfeb3f74eef802de32f6f485512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "619773ca0df3492e9d95c4052c203b41" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 40183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:10.048851 19427 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:10:10.049218 19427 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:10:10.049274 19427 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:10:10.049671 19422 placement_policy_util.cc:407] tablet 54dde18516574ee9a3c02dc730e926e7: detected majority of replicas (2 of 3) at location L0
I20240501 14:10:11.051405 19422 placement_policy_util.cc:407] tablet 54dde18516574ee9a3c02dc730e926e7: detected majority of replicas (2 of 3) at location L0
I20240501 14:10:12.049574 19427 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:10:12.049907 19427 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:10:12.049976 19427 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:10:12.052995 19422 placement_policy_util.cc:407] tablet 54dde18516574ee9a3c02dc730e926e7: detected majority of replicas (2 of 3) at location L0
I20240501 14:10:13.054575 19422 placement_policy_util.cc:407] tablet 54dde18516574ee9a3c02dc730e926e7: detected majority of replicas (2 of 3) at location L0
I20240501 14:10:14.050285 19427 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:10:14.050630 19427 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:10:14.050695 19427 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:10:14.056283 19422 placement_policy_util.cc:407] tablet 54dde18516574ee9a3c02dc730e926e7: detected majority of replicas (2 of 3) at location L0
I20240501 14:10:15.058254 19422 placement_policy_util.cc:407] tablet 54dde18516574ee9a3c02dc730e926e7: detected majority of replicas (2 of 3) at location L0
I20240501 14:10:16.051002 19427 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:10:16.051399 19427 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:10:16.051468 19427 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:10:16.060472 19422 placement_policy_util.cc:407] tablet 54dde18516574ee9a3c02dc730e926e7: detected majority of replicas (2 of 3) at location L0
I20240501 14:10:17.062170 19422 placement_policy_util.cc:407] tablet 54dde18516574ee9a3c02dc730e926e7: detected majority of replicas (2 of 3) at location L0
I20240501 14:10:18.051753 19427 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:10:18.052139 19427 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:10:18.052218 19427 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:10:18.063854 19422 placement_policy_util.cc:407] tablet 54dde18516574ee9a3c02dc730e926e7: detected majority of replicas (2 of 3) at location L0
I20240501 14:10:19.065495 19422 placement_policy_util.cc:407] tablet 54dde18516574ee9a3c02dc730e926e7: detected majority of replicas (2 of 3) at location L0
I20240501 14:10:20.052579 19427 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:10:20.052904 19427 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:10:20.052969 19427 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:10:20.067193 19422 placement_policy_util.cc:407] tablet 54dde18516574ee9a3c02dc730e926e7: detected majority of replicas (2 of 3) at location L0
I20240501 14:10:20.630661 16439 tablet_server.cc:178] TabletServer@127.16.13.193:0 shutting down...
I20240501 14:10:20.634327 16439 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:10:20.634554 16439 tablet_replica.cc:330] T 54dde18516574ee9a3c02dc730e926e7 P 07c81dfeb3f74eef802de32f6f485512: stopping tablet replica
I20240501 14:10:20.634697 16439 raft_consensus.cc:2229] T 54dde18516574ee9a3c02dc730e926e7 P 07c81dfeb3f74eef802de32f6f485512 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:10:20.634917 16439 raft_consensus.cc:2258] T 54dde18516574ee9a3c02dc730e926e7 P 07c81dfeb3f74eef802de32f6f485512 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:20.648102 16439 tablet_server.cc:194] TabletServer@127.16.13.193:0 shutdown complete.
I20240501 14:10:20.650064 16439 tablet_server.cc:178] TabletServer@127.16.13.194:0 shutting down...
I20240501 14:10:20.654338 16439 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:10:20.654515 16439 tablet_replica.cc:330] T 54dde18516574ee9a3c02dc730e926e7 P 619773ca0df3492e9d95c4052c203b41: stopping tablet replica
I20240501 14:10:20.654623 16439 raft_consensus.cc:2229] T 54dde18516574ee9a3c02dc730e926e7 P 619773ca0df3492e9d95c4052c203b41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:20.654722 16439 raft_consensus.cc:2258] T 54dde18516574ee9a3c02dc730e926e7 P 619773ca0df3492e9d95c4052c203b41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:20.667290 16439 tablet_server.cc:194] TabletServer@127.16.13.194:0 shutdown complete.
I20240501 14:10:20.669418 16439 tablet_server.cc:178] TabletServer@127.16.13.195:0 shutting down...
I20240501 14:10:20.673463 16439 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:10:20.673643 16439 tablet_replica.cc:330] T 54dde18516574ee9a3c02dc730e926e7 P 4cde0e47c95341cbae7fde5be0d3aee7: stopping tablet replica
I20240501 14:10:20.673759 16439 raft_consensus.cc:2229] T 54dde18516574ee9a3c02dc730e926e7 P 4cde0e47c95341cbae7fde5be0d3aee7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:20.673866 16439 raft_consensus.cc:2258] T 54dde18516574ee9a3c02dc730e926e7 P 4cde0e47c95341cbae7fde5be0d3aee7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:20.687091 16439 tablet_server.cc:194] TabletServer@127.16.13.195:0 shutdown complete.
I20240501 14:10:20.689110 16439 master.cc:537] Master@127.16.13.254:39135 shutting down...
I20240501 14:10:20.692729 16439 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b3d6369ed2fb496f8825d34ae65316f5 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:10:20.692906 16439 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b3d6369ed2fb496f8825d34ae65316f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:20.692968 16439 tablet_replica.cc:330] T 00000000000000000000000000000000 P b3d6369ed2fb496f8825d34ae65316f5: stopping tablet replica
I20240501 14:10:20.705114 16439 master.cc:552] Master@127.16.13.254:39135 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (12698 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
I20240501 14:10:20.711553 16439 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.13.254:40095
I20240501 14:10:20.711867 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:10:20.714141 19696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:20.714890 19697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:20.715672 19699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:10:20.716405 16439 server_base.cc:1034] running on GCE node
I20240501 14:10:20.716607 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:10:20.716645 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:10:20.716661 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572620716661 us; error 0 us; skew 500 ppm
I20240501 14:10:20.717592 16439 webserver.cc:460] Webserver started at http://127.16.13.254:45401/ using document root <none> and password file <none>
I20240501 14:10:20.717746 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:10:20.717790 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:10:20.717876 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:10:20.718284 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/master-0-root/instance:
uuid: "3527bff3ee92408f84e04d7b341a38cd"
format_stamp: "Formatted at 2024-05-01 14:10:20 on dist-test-slave-mgxc"
I20240501 14:10:20.719687 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 14:10:20.720664 19704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:20.720935 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:20.721004 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/master-0-root
uuid: "3527bff3ee92408f84e04d7b341a38cd"
format_stamp: "Formatted at 2024-05-01 14:10:20 on dist-test-slave-mgxc"
I20240501 14:10:20.721083 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:10:20.729694 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:10:20.730027 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:10:20.733899 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.254:40095
I20240501 14:10:20.735824 19756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.254:40095 every 8 connection(s)
I20240501 14:10:20.736361 19757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:20.738137 19757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3527bff3ee92408f84e04d7b341a38cd: Bootstrap starting.
I20240501 14:10:20.738889 19757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3527bff3ee92408f84e04d7b341a38cd: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:20.739884 19757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3527bff3ee92408f84e04d7b341a38cd: No bootstrap required, opened a new log
I20240501 14:10:20.740253 19757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3527bff3ee92408f84e04d7b341a38cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3527bff3ee92408f84e04d7b341a38cd" member_type: VOTER }
I20240501 14:10:20.740340 19757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3527bff3ee92408f84e04d7b341a38cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:20.740361 19757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3527bff3ee92408f84e04d7b341a38cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3527bff3ee92408f84e04d7b341a38cd, State: Initialized, Role: FOLLOWER
I20240501 14:10:20.740476 19757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3527bff3ee92408f84e04d7b341a38cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3527bff3ee92408f84e04d7b341a38cd" member_type: VOTER }
I20240501 14:10:20.740559 19757 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3527bff3ee92408f84e04d7b341a38cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:10:20.740598 19757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3527bff3ee92408f84e04d7b341a38cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:10:20.740634 19757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3527bff3ee92408f84e04d7b341a38cd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:20.741313 19757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3527bff3ee92408f84e04d7b341a38cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3527bff3ee92408f84e04d7b341a38cd" member_type: VOTER }
I20240501 14:10:20.741438 19757 leader_election.cc:302] T 00000000000000000000000000000000 P 3527bff3ee92408f84e04d7b341a38cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3527bff3ee92408f84e04d7b341a38cd; no voters:
I20240501 14:10:20.741595 19757 leader_election.cc:288] T 00000000000000000000000000000000 P 3527bff3ee92408f84e04d7b341a38cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:10:20.741726 19760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3527bff3ee92408f84e04d7b341a38cd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:20.741925 19760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3527bff3ee92408f84e04d7b341a38cd [term 1 LEADER]: Becoming Leader. State: Replica: 3527bff3ee92408f84e04d7b341a38cd, State: Running, Role: LEADER
I20240501 14:10:20.742043 19757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3527bff3ee92408f84e04d7b341a38cd [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:10:20.742054 19760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3527bff3ee92408f84e04d7b341a38cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3527bff3ee92408f84e04d7b341a38cd" member_type: VOTER }
I20240501 14:10:20.742532 19761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3527bff3ee92408f84e04d7b341a38cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3527bff3ee92408f84e04d7b341a38cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3527bff3ee92408f84e04d7b341a38cd" member_type: VOTER } }
I20240501 14:10:20.742708 19761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3527bff3ee92408f84e04d7b341a38cd [sys.catalog]: This master's current role is: LEADER
I20240501 14:10:20.742578 19762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3527bff3ee92408f84e04d7b341a38cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3527bff3ee92408f84e04d7b341a38cd. Latest consensus state: current_term: 1 leader_uuid: "3527bff3ee92408f84e04d7b341a38cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3527bff3ee92408f84e04d7b341a38cd" member_type: VOTER } }
I20240501 14:10:20.742988 19762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3527bff3ee92408f84e04d7b341a38cd [sys.catalog]: This master's current role is: LEADER
I20240501 14:10:20.743443 19767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:10:20.744352 19767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:10:20.744549 16439 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 14:10:20.746122 19767 catalog_manager.cc:1347] Generated new cluster ID: 36108337bef04864bb66819cf40402a7
I20240501 14:10:20.746181 19767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:10:20.763832 19767 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:10:20.764446 19767 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:10:20.775125 19767 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3527bff3ee92408f84e04d7b341a38cd: Generated new TSK 0
I20240501 14:10:20.775310 19767 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:10:20.776862 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:10:20.779430 19779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:20.780239 19781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:20.780927 19778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:10:20.781750 16439 server_base.cc:1034] running on GCE node
I20240501 14:10:20.781914 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:10:20.781953 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:10:20.781976 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572620781976 us; error 0 us; skew 500 ppm
I20240501 14:10:20.782917 16439 webserver.cc:460] Webserver started at http://127.16.13.193:46043/ using document root <none> and password file <none>
I20240501 14:10:20.783049 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:10:20.783089 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:10:20.783150 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:10:20.783504 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-0-root/instance:
uuid: "7d4b038952284c81a07423e42a349192"
format_stamp: "Formatted at 2024-05-01 14:10:20 on dist-test-slave-mgxc"
I20240501 14:10:20.784994 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 14:10:20.785938 19786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:20.786190 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:20.786257 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-0-root
uuid: "7d4b038952284c81a07423e42a349192"
format_stamp: "Formatted at 2024-05-01 14:10:20 on dist-test-slave-mgxc"
I20240501 14:10:20.786334 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:10:20.797562 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:10:20.797979 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:10:20.798290 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:10:20.798760 16439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:10:20.798800 16439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:20.798847 16439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:10:20.798880 16439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:20.802971 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.193:38811
I20240501 14:10:20.803001 19849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.193:38811 every 8 connection(s)
I20240501 14:10:20.803805 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:10:20.806332 19854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:20.807168 19855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:20.807984 19857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:10:20.808717 16439 server_base.cc:1034] running on GCE node
I20240501 14:10:20.808786 19850 heartbeater.cc:347] Connected to a master server at 127.16.13.254:40095
I20240501 14:10:20.808856 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:10:20.808881 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:10:20.808889 19850 heartbeater.cc:464] Registering TS with master...
I20240501 14:10:20.808897 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572620808897 us; error 0 us; skew 500 ppm
I20240501 14:10:20.809128 19850 heartbeater.cc:511] Master 127.16.13.254:40095 requested a full tablet report, sending...
I20240501 14:10:20.809873 19721 ts_manager.cc:194] Registered new tserver with Master: 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811)
I20240501 14:10:20.810071 16439 webserver.cc:460] Webserver started at http://127.16.13.194:39775/ using document root <none> and password file <none>
I20240501 14:10:20.810227 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:10:20.810289 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:10:20.810357 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:10:20.810806 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-1-root/instance:
uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
format_stamp: "Formatted at 2024-05-01 14:10:20 on dist-test-slave-mgxc"
I20240501 14:10:20.810910 19721 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:32910
I20240501 14:10:20.812474 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 14:10:20.813328 19862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:20.813608 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:20.813673 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-1-root
uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
format_stamp: "Formatted at 2024-05-01 14:10:20 on dist-test-slave-mgxc"
I20240501 14:10:20.813732 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:10:20.830942 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:10:20.831369 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:10:20.831648 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:10:20.832218 16439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:10:20.832260 16439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:20.832299 16439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:10:20.832332 16439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:20.836380 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.194:39591
I20240501 14:10:20.836426 19925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.194:39591 every 8 connection(s)
I20240501 14:10:20.837320 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:10:20.839905 19930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:20.840615 19929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:10:20.841347 19926 heartbeater.cc:347] Connected to a master server at 127.16.13.254:40095
I20240501 14:10:20.841444 19926 heartbeater.cc:464] Registering TS with master...
W20240501 14:10:20.841492 19932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:10:20.841686 19926 heartbeater.cc:511] Master 127.16.13.254:40095 requested a full tablet report, sending...
I20240501 14:10:20.842228 16439 server_base.cc:1034] running on GCE node
I20240501 14:10:20.842413 19721 ts_manager.cc:194] Registered new tserver with Master: c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591)
I20240501 14:10:20.842425 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:10:20.842476 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:10:20.842491 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572620842491 us; error 0 us; skew 500 ppm
I20240501 14:10:20.843194 19721 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:32918
I20240501 14:10:20.843705 16439 webserver.cc:460] Webserver started at http://127.16.13.195:44145/ using document root <none> and password file <none>
I20240501 14:10:20.843875 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:10:20.843955 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:10:20.844034 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:10:20.844431 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-2-root/instance:
uuid: "94100faea1434b27a8c6d129d1b562bc"
format_stamp: "Formatted at 2024-05-01 14:10:20 on dist-test-slave-mgxc"
I20240501 14:10:20.845780 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 14:10:20.846639 19937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:20.846860 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:20.846920 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-2-root
uuid: "94100faea1434b27a8c6d129d1b562bc"
format_stamp: "Formatted at 2024-05-01 14:10:20 on dist-test-slave-mgxc"
I20240501 14:10:20.846997 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:10:20.863363 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:10:20.863775 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:10:20.864102 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:10:20.864578 16439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:10:20.864615 16439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:20.864660 16439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:10:20.864691 16439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:20.869076 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.195:36685
I20240501 14:10:20.869171 20000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.195:36685 every 8 connection(s)
I20240501 14:10:20.873905 20001 heartbeater.cc:347] Connected to a master server at 127.16.13.254:40095
I20240501 14:10:20.874004 20001 heartbeater.cc:464] Registering TS with master...
I20240501 14:10:20.874254 20001 heartbeater.cc:511] Master 127.16.13.254:40095 requested a full tablet report, sending...
I20240501 14:10:20.874814 19721 ts_manager.cc:194] Registered new tserver with Master: 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:20.875088 16439 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005568687s
I20240501 14:10:20.875448 19721 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:32932
I20240501 14:10:21.811949 19850 heartbeater.cc:503] Master 127.16.13.254:40095 was elected leader, sending a full tablet report...
I20240501 14:10:21.844228 19926 heartbeater.cc:503] Master 127.16.13.254:40095 was elected leader, sending a full tablet report...
I20240501 14:10:21.876401 20001 heartbeater.cc:503] Master 127.16.13.254:40095 was elected leader, sending a full tablet report...
I20240501 14:10:21.901918 16439 test_util.cc:268] Using random seed: 202502233
I20240501 14:10:21.915040 19721 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56450:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu""\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 14:10:21.915608 19721 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 14:10:21.929955 19964 tablet_service.cc:1467] Processing CreateTablet for tablet 603909ab31c7462c9e2c893184a5fc6c (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240501 14:10:21.930145 19959 tablet_service.cc:1467] Processing CreateTablet for tablet 57d657b87203496ca9d47d00fd98c1af (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240501 14:10:21.930358 19964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 603909ab31c7462c9e2c893184a5fc6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.930553 19957 tablet_service.cc:1467] Processing CreateTablet for tablet ffd3ce2e5a1a4a0182e578d471a022ae (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240501 14:10:21.930817 19963 tablet_service.cc:1467] Processing CreateTablet for tablet a47e39994ef3497e96bedd8a36133e00 (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240501 14:10:21.930864 19811 tablet_service.cc:1467] Processing CreateTablet for tablet a726c9cea76241df8a00d52d6175db7a (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240501 14:10:21.931152 19811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a726c9cea76241df8a00d52d6175db7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.931185 19965 tablet_service.cc:1467] Processing CreateTablet for tablet 515893894ce0478caa58378e6c2e047b (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240501 14:10:21.931273 19962 tablet_service.cc:1467] Processing CreateTablet for tablet a726c9cea76241df8a00d52d6175db7a (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240501 14:10:21.931716 19885 tablet_service.cc:1467] Processing CreateTablet for tablet 352498cc20094372a957c030cebf375e (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240501 14:10:21.931934 19960 tablet_service.cc:1467] Processing CreateTablet for tablet 352498cc20094372a957c030cebf375e (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240501 14:10:21.932010 19885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 352498cc20094372a957c030cebf375e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.932183 19890 tablet_service.cc:1467] Processing CreateTablet for tablet 515893894ce0478caa58378e6c2e047b (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240501 14:10:21.932384 19958 tablet_service.cc:1467] Processing CreateTablet for tablet f796e2ca050e4666b9eb2dfd8a9ee2ff (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240501 14:10:21.932605 19889 tablet_service.cc:1467] Processing CreateTablet for tablet 603909ab31c7462c9e2c893184a5fc6c (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240501 14:10:21.932862 19956 tablet_service.cc:1467] Processing CreateTablet for tablet 9d0888af56f2427c9387c3682f211a58 (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240501 14:10:21.933001 19888 tablet_service.cc:1467] Processing CreateTablet for tablet a47e39994ef3497e96bedd8a36133e00 (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240501 14:10:21.933270 19814 tablet_service.cc:1467] Processing CreateTablet for tablet 515893894ce0478caa58378e6c2e047b (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240501 14:10:21.933385 19887 tablet_service.cc:1467] Processing CreateTablet for tablet a726c9cea76241df8a00d52d6175db7a (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240501 14:10:21.933499 19814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 515893894ce0478caa58378e6c2e047b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.933779 19886 tablet_service.cc:1467] Processing CreateTablet for tablet 2e6457de95c64fb19ae1d294c436c8f7 (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240501 14:10:21.934152 19884 tablet_service.cc:1467] Processing CreateTablet for tablet 57d657b87203496ca9d47d00fd98c1af (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240501 14:10:21.930570 19959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57d657b87203496ca9d47d00fd98c1af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.934592 19883 tablet_service.cc:1467] Processing CreateTablet for tablet f796e2ca050e4666b9eb2dfd8a9ee2ff (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240501 14:10:21.934785 19956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d0888af56f2427c9387c3682f211a58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.934988 19882 tablet_service.cc:1467] Processing CreateTablet for tablet ffd3ce2e5a1a4a0182e578d471a022ae (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240501 14:10:21.935407 19881 tablet_service.cc:1467] Processing CreateTablet for tablet 9d0888af56f2427c9387c3682f211a58 (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240501 14:10:21.935813 19813 tablet_service.cc:1467] Processing CreateTablet for tablet 603909ab31c7462c9e2c893184a5fc6c (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240501 14:10:21.935928 19957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffd3ce2e5a1a4a0182e578d471a022ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.936054 19813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 603909ab31c7462c9e2c893184a5fc6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.936759 19965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 515893894ce0478caa58378e6c2e047b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.932397 19890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 515893894ce0478caa58378e6c2e047b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.938040 19812 tablet_service.cc:1467] Processing CreateTablet for tablet a47e39994ef3497e96bedd8a36133e00 (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240501 14:10:21.938123 19886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e6457de95c64fb19ae1d294c436c8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.938257 19812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a47e39994ef3497e96bedd8a36133e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.938257 19884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57d657b87203496ca9d47d00fd98c1af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.938480 19888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a47e39994ef3497e96bedd8a36133e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.938882 19887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a726c9cea76241df8a00d52d6175db7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.940016 19889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 603909ab31c7462c9e2c893184a5fc6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.940557 19882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffd3ce2e5a1a4a0182e578d471a022ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.940766 19881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d0888af56f2427c9387c3682f211a58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.940965 19883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f796e2ca050e4666b9eb2dfd8a9ee2ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.931722 19961 tablet_service.cc:1467] Processing CreateTablet for tablet 2e6457de95c64fb19ae1d294c436c8f7 (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240501 14:10:21.941774 19961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e6457de95c64fb19ae1d294c436c8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.942077 20023 tablet_bootstrap.cc:492] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1: Bootstrap starting.
I20240501 14:10:21.942268 19890 tablet_service.cc:1467] Processing CreateTablet for tablet 4b28da769e1f46e1b05bd25e8428316f (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240501 14:10:21.942488 19890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b28da769e1f46e1b05bd25e8428316f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.943053 19809 tablet_service.cc:1467] Processing CreateTablet for tablet 352498cc20094372a957c030cebf375e (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240501 14:10:21.943310 19809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 352498cc20094372a957c030cebf375e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.943454 20023 tablet_bootstrap.cc:654] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:21.943776 19807 tablet_service.cc:1467] Processing CreateTablet for tablet f796e2ca050e4666b9eb2dfd8a9ee2ff (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240501 14:10:21.944036 19807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f796e2ca050e4666b9eb2dfd8a9ee2ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.945638 19890 tablet_service.cc:1467] Processing CreateTablet for tablet b8922efb08b3491fa8d947793206bf7a (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240501 14:10:21.945858 19890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8922efb08b3491fa8d947793206bf7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.945894 19810 tablet_service.cc:1467] Processing CreateTablet for tablet 2e6457de95c64fb19ae1d294c436c8f7 (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240501 14:10:21.946108 19810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e6457de95c64fb19ae1d294c436c8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.947410 19808 tablet_service.cc:1467] Processing CreateTablet for tablet 57d657b87203496ca9d47d00fd98c1af (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240501 14:10:21.947649 19808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57d657b87203496ca9d47d00fd98c1af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.948006 19958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f796e2ca050e4666b9eb2dfd8a9ee2ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.948928 19806 tablet_service.cc:1467] Processing CreateTablet for tablet ffd3ce2e5a1a4a0182e578d471a022ae (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240501 14:10:21.949169 19806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffd3ce2e5a1a4a0182e578d471a022ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.949550 19805 tablet_service.cc:1467] Processing CreateTablet for tablet 9d0888af56f2427c9387c3682f211a58 (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240501 14:10:21.949801 19805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d0888af56f2427c9387c3682f211a58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.949949 19962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a726c9cea76241df8a00d52d6175db7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.951892 19960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 352498cc20094372a957c030cebf375e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.952106 20023 tablet_bootstrap.cc:492] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1: No bootstrap required, opened a new log
I20240501 14:10:21.952234 20023 ts_tablet_manager.cc:1367] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:10:21.952733 20023 raft_consensus.cc:348] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.952953 20023 raft_consensus.cc:374] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:21.952972 20021 tablet_bootstrap.cc:492] T a726c9cea76241df8a00d52d6175db7a P 7d4b038952284c81a07423e42a349192: Bootstrap starting.
I20240501 14:10:21.952999 20023 raft_consensus.cc:729] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d2a3d7c38d4794a1ee78eaef2940d1, State: Initialized, Role: FOLLOWER
I20240501 14:10:21.953152 20023 consensus_queue.cc:260] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.953634 20023 ts_tablet_manager.cc:1398] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:21.953758 19963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a47e39994ef3497e96bedd8a36133e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.953795 20023 tablet_bootstrap.cc:492] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1: Bootstrap starting.
I20240501 14:10:21.953974 20021 tablet_bootstrap.cc:654] T a726c9cea76241df8a00d52d6175db7a P 7d4b038952284c81a07423e42a349192: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:21.955121 20021 tablet_bootstrap.cc:492] T a726c9cea76241df8a00d52d6175db7a P 7d4b038952284c81a07423e42a349192: No bootstrap required, opened a new log
I20240501 14:10:21.955211 20021 ts_tablet_manager.cc:1367] T a726c9cea76241df8a00d52d6175db7a P 7d4b038952284c81a07423e42a349192: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:21.955627 20021 raft_consensus.cc:348] T a726c9cea76241df8a00d52d6175db7a P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.955725 20021 raft_consensus.cc:374] T a726c9cea76241df8a00d52d6175db7a P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:21.955763 20021 raft_consensus.cc:729] T a726c9cea76241df8a00d52d6175db7a P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d4b038952284c81a07423e42a349192, State: Initialized, Role: FOLLOWER
I20240501 14:10:21.955814 20023 tablet_bootstrap.cc:654] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:21.955868 20021 consensus_queue.cc:260] T a726c9cea76241df8a00d52d6175db7a P 7d4b038952284c81a07423e42a349192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.956528 20021 ts_tablet_manager.cc:1398] T a726c9cea76241df8a00d52d6175db7a P 7d4b038952284c81a07423e42a349192: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:21.956740 20021 tablet_bootstrap.cc:492] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192: Bootstrap starting.
I20240501 14:10:21.957638 20021 tablet_bootstrap.cc:654] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:21.958436 19805 tablet_service.cc:1467] Processing CreateTablet for tablet 4b28da769e1f46e1b05bd25e8428316f (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240501 14:10:21.958513 19806 tablet_service.cc:1467] Processing CreateTablet for tablet b8922efb08b3491fa8d947793206bf7a (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240501 14:10:21.958665 19805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b28da769e1f46e1b05bd25e8428316f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.958912 19964 tablet_service.cc:1467] Processing CreateTablet for tablet 4b28da769e1f46e1b05bd25e8428316f (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240501 14:10:21.959005 19963 tablet_service.cc:1467] Processing CreateTablet for tablet b8922efb08b3491fa8d947793206bf7a (DEFAULT_TABLE table=test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240501 14:10:21.959134 19964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b28da769e1f46e1b05bd25e8428316f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.959283 19963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8922efb08b3491fa8d947793206bf7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.959673 19806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8922efb08b3491fa8d947793206bf7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:21.960443 20023 tablet_bootstrap.cc:492] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1: No bootstrap required, opened a new log
I20240501 14:10:21.960520 20023 ts_tablet_manager.cc:1367] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:10:21.960971 20023 raft_consensus.cc:348] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.961076 20023 raft_consensus.cc:374] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:21.961107 20023 raft_consensus.cc:729] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d2a3d7c38d4794a1ee78eaef2940d1, State: Initialized, Role: FOLLOWER
I20240501 14:10:21.961197 20022 tablet_bootstrap.cc:492] T 57d657b87203496ca9d47d00fd98c1af P 94100faea1434b27a8c6d129d1b562bc: Bootstrap starting.
I20240501 14:10:21.961227 20023 consensus_queue.cc:260] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.961517 20023 ts_tablet_manager.cc:1398] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:21.961694 20023 tablet_bootstrap.cc:492] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1: Bootstrap starting.
I20240501 14:10:21.962184 20022 tablet_bootstrap.cc:654] T 57d657b87203496ca9d47d00fd98c1af P 94100faea1434b27a8c6d129d1b562bc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:21.962595 20023 tablet_bootstrap.cc:654] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:21.964015 20023 tablet_bootstrap.cc:492] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1: No bootstrap required, opened a new log
I20240501 14:10:21.964015 20021 tablet_bootstrap.cc:492] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192: No bootstrap required, opened a new log
I20240501 14:10:21.964141 20021 ts_tablet_manager.cc:1367] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:10:21.964143 20023 ts_tablet_manager.cc:1367] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:21.964629 20021 raft_consensus.cc:348] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.964730 20021 raft_consensus.cc:374] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:21.964764 20023 raft_consensus.cc:348] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.964802 20022 tablet_bootstrap.cc:492] T 57d657b87203496ca9d47d00fd98c1af P 94100faea1434b27a8c6d129d1b562bc: No bootstrap required, opened a new log
I20240501 14:10:21.964794 20021 raft_consensus.cc:729] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d4b038952284c81a07423e42a349192, State: Initialized, Role: FOLLOWER
I20240501 14:10:21.964856 20023 raft_consensus.cc:374] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:21.964897 20023 raft_consensus.cc:729] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d2a3d7c38d4794a1ee78eaef2940d1, State: Initialized, Role: FOLLOWER
I20240501 14:10:21.964917 20022 ts_tablet_manager.cc:1367] T 57d657b87203496ca9d47d00fd98c1af P 94100faea1434b27a8c6d129d1b562bc: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:10:21.965006 20021 consensus_queue.cc:260] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.965294 20021 ts_tablet_manager.cc:1398] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:21.965250 20023 consensus_queue.cc:260] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.965305 20022 raft_consensus.cc:348] T 57d657b87203496ca9d47d00fd98c1af P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.965476 20022 raft_consensus.cc:374] T 57d657b87203496ca9d47d00fd98c1af P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:21.965523 20022 raft_consensus.cc:729] T 57d657b87203496ca9d47d00fd98c1af P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94100faea1434b27a8c6d129d1b562bc, State: Initialized, Role: FOLLOWER
I20240501 14:10:21.965646 20022 consensus_queue.cc:260] T 57d657b87203496ca9d47d00fd98c1af P 94100faea1434b27a8c6d129d1b562bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.965759 20021 tablet_bootstrap.cc:492] T 603909ab31c7462c9e2c893184a5fc6c P 7d4b038952284c81a07423e42a349192: Bootstrap starting.
I20240501 14:10:21.965934 20022 ts_tablet_manager.cc:1398] T 57d657b87203496ca9d47d00fd98c1af P 94100faea1434b27a8c6d129d1b562bc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:21.966105 20022 tablet_bootstrap.cc:492] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc: Bootstrap starting.
I20240501 14:10:21.966714 20021 tablet_bootstrap.cc:654] T 603909ab31c7462c9e2c893184a5fc6c P 7d4b038952284c81a07423e42a349192: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:21.966809 20022 tablet_bootstrap.cc:654] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:21.968029 20022 tablet_bootstrap.cc:492] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc: No bootstrap required, opened a new log
I20240501 14:10:21.968123 20022 ts_tablet_manager.cc:1367] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:21.968310 20023 ts_tablet_manager.cc:1398] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:10:21.968577 20022 raft_consensus.cc:348] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.968755 20022 raft_consensus.cc:374] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:21.968824 20022 raft_consensus.cc:729] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94100faea1434b27a8c6d129d1b562bc, State: Initialized, Role: FOLLOWER
I20240501 14:10:21.968955 20022 consensus_queue.cc:260] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.969095 20023 tablet_bootstrap.cc:492] T 603909ab31c7462c9e2c893184a5fc6c P c4d2a3d7c38d4794a1ee78eaef2940d1: Bootstrap starting.
I20240501 14:10:21.969278 20022 ts_tablet_manager.cc:1398] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:21.969468 20022 tablet_bootstrap.cc:492] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc: Bootstrap starting.
I20240501 14:10:21.970151 20023 tablet_bootstrap.cc:654] T 603909ab31c7462c9e2c893184a5fc6c P c4d2a3d7c38d4794a1ee78eaef2940d1: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:21.970279 20022 tablet_bootstrap.cc:654] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:21.971298 20022 tablet_bootstrap.cc:492] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc: No bootstrap required, opened a new log
I20240501 14:10:21.971392 20022 ts_tablet_manager.cc:1367] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:21.971395 20023 tablet_bootstrap.cc:492] T 603909ab31c7462c9e2c893184a5fc6c P c4d2a3d7c38d4794a1ee78eaef2940d1: No bootstrap required, opened a new log
I20240501 14:10:21.971536 20023 ts_tablet_manager.cc:1367] T 603909ab31c7462c9e2c893184a5fc6c P c4d2a3d7c38d4794a1ee78eaef2940d1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:21.971801 20022 raft_consensus.cc:348] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.971889 20022 raft_consensus.cc:374] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:21.971897 20023 raft_consensus.cc:348] T 603909ab31c7462c9e2c893184a5fc6c P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.972034 20022 raft_consensus.cc:729] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94100faea1434b27a8c6d129d1b562bc, State: Initialized, Role: FOLLOWER
I20240501 14:10:21.972208 20022 consensus_queue.cc:260] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.972509 20021 tablet_bootstrap.cc:492] T 603909ab31c7462c9e2c893184a5fc6c P 7d4b038952284c81a07423e42a349192: No bootstrap required, opened a new log
I20240501 14:10:21.972600 20021 ts_tablet_manager.cc:1367] T 603909ab31c7462c9e2c893184a5fc6c P 7d4b038952284c81a07423e42a349192: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:10:21.972651 20022 ts_tablet_manager.cc:1398] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:21.972837 20022 tablet_bootstrap.cc:492] T 515893894ce0478caa58378e6c2e047b P 94100faea1434b27a8c6d129d1b562bc: Bootstrap starting.
I20240501 14:10:21.973039 20021 raft_consensus.cc:348] T 603909ab31c7462c9e2c893184a5fc6c P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.973161 20021 raft_consensus.cc:374] T 603909ab31c7462c9e2c893184a5fc6c P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:21.973202 20021 raft_consensus.cc:729] T 603909ab31c7462c9e2c893184a5fc6c P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d4b038952284c81a07423e42a349192, State: Initialized, Role: FOLLOWER
I20240501 14:10:21.973348 20021 consensus_queue.cc:260] T 603909ab31c7462c9e2c893184a5fc6c P 7d4b038952284c81a07423e42a349192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.973659 20021 ts_tablet_manager.cc:1398] T 603909ab31c7462c9e2c893184a5fc6c P 7d4b038952284c81a07423e42a349192: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:21.973853 20021 tablet_bootstrap.cc:492] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192: Bootstrap starting.
I20240501 14:10:21.974443 20022 tablet_bootstrap.cc:654] T 515893894ce0478caa58378e6c2e047b P 94100faea1434b27a8c6d129d1b562bc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:21.974889 20021 tablet_bootstrap.cc:654] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:21.975085 20023 raft_consensus.cc:374] T 603909ab31c7462c9e2c893184a5fc6c P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:21.975152 20023 raft_consensus.cc:729] T 603909ab31c7462c9e2c893184a5fc6c P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d2a3d7c38d4794a1ee78eaef2940d1, State: Initialized, Role: FOLLOWER
I20240501 14:10:21.975303 20023 consensus_queue.cc:260] T 603909ab31c7462c9e2c893184a5fc6c P c4d2a3d7c38d4794a1ee78eaef2940d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.975859 20022 tablet_bootstrap.cc:492] T 515893894ce0478caa58378e6c2e047b P 94100faea1434b27a8c6d129d1b562bc: No bootstrap required, opened a new log
I20240501 14:10:21.976011 20022 ts_tablet_manager.cc:1367] T 515893894ce0478caa58378e6c2e047b P 94100faea1434b27a8c6d129d1b562bc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240501 14:10:21.976056 20021 tablet_bootstrap.cc:492] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192: No bootstrap required, opened a new log
I20240501 14:10:21.976122 20021 ts_tablet_manager.cc:1367] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:21.976469 20022 raft_consensus.cc:348] T 515893894ce0478caa58378e6c2e047b P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.976488 20021 raft_consensus.cc:348] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.976627 20022 raft_consensus.cc:374] T 515893894ce0478caa58378e6c2e047b P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:21.976637 20021 raft_consensus.cc:374] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:21.976676 20022 raft_consensus.cc:729] T 515893894ce0478caa58378e6c2e047b P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94100faea1434b27a8c6d129d1b562bc, State: Initialized, Role: FOLLOWER
I20240501 14:10:21.976692 20021 raft_consensus.cc:729] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d4b038952284c81a07423e42a349192, State: Initialized, Role: FOLLOWER
I20240501 14:10:21.976807 20021 consensus_queue.cc:260] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.976807 20022 consensus_queue.cc:260] T 515893894ce0478caa58378e6c2e047b P 94100faea1434b27a8c6d129d1b562bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.977128 20021 ts_tablet_manager.cc:1398] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:21.977190 20022 ts_tablet_manager.cc:1398] T 515893894ce0478caa58378e6c2e047b P 94100faea1434b27a8c6d129d1b562bc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:21.977687 20021 tablet_bootstrap.cc:492] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192: Bootstrap starting.
I20240501 14:10:21.977989 20022 tablet_bootstrap.cc:492] T 2e6457de95c64fb19ae1d294c436c8f7 P 94100faea1434b27a8c6d129d1b562bc: Bootstrap starting.
I20240501 14:10:21.978691 20021 tablet_bootstrap.cc:654] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:21.978950 20022 tablet_bootstrap.cc:654] T 2e6457de95c64fb19ae1d294c436c8f7 P 94100faea1434b27a8c6d129d1b562bc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:21.979722 20023 ts_tablet_manager.cc:1398] T 603909ab31c7462c9e2c893184a5fc6c P c4d2a3d7c38d4794a1ee78eaef2940d1: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20240501 14:10:21.980360 20021 tablet_bootstrap.cc:492] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192: No bootstrap required, opened a new log
I20240501 14:10:21.980453 20021 ts_tablet_manager.cc:1367] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:10:21.980950 20021 raft_consensus.cc:348] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.981099 20021 raft_consensus.cc:374] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:21.981160 20021 raft_consensus.cc:729] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d4b038952284c81a07423e42a349192, State: Initialized, Role: FOLLOWER
I20240501 14:10:21.981293 20021 consensus_queue.cc:260] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.981379 20023 tablet_bootstrap.cc:492] T ffd3ce2e5a1a4a0182e578d471a022ae P c4d2a3d7c38d4794a1ee78eaef2940d1: Bootstrap starting.
I20240501 14:10:21.981583 20021 ts_tablet_manager.cc:1398] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:21.981776 20021 tablet_bootstrap.cc:492] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192: Bootstrap starting.
I20240501 14:10:21.982535 20022 tablet_bootstrap.cc:492] T 2e6457de95c64fb19ae1d294c436c8f7 P 94100faea1434b27a8c6d129d1b562bc: No bootstrap required, opened a new log
I20240501 14:10:21.982623 20022 ts_tablet_manager.cc:1367] T 2e6457de95c64fb19ae1d294c436c8f7 P 94100faea1434b27a8c6d129d1b562bc: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:10:21.982780 20021 tablet_bootstrap.cc:654] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:21.983213 20022 raft_consensus.cc:348] T 2e6457de95c64fb19ae1d294c436c8f7 P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.983331 20022 raft_consensus.cc:374] T 2e6457de95c64fb19ae1d294c436c8f7 P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:21.983369 20022 raft_consensus.cc:729] T 2e6457de95c64fb19ae1d294c436c8f7 P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94100faea1434b27a8c6d129d1b562bc, State: Initialized, Role: FOLLOWER
I20240501 14:10:21.983510 20022 consensus_queue.cc:260] T 2e6457de95c64fb19ae1d294c436c8f7 P 94100faea1434b27a8c6d129d1b562bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.983965 20023 tablet_bootstrap.cc:654] T ffd3ce2e5a1a4a0182e578d471a022ae P c4d2a3d7c38d4794a1ee78eaef2940d1: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:21.984097 20021 tablet_bootstrap.cc:492] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192: No bootstrap required, opened a new log
I20240501 14:10:21.984179 20021 ts_tablet_manager.cc:1367] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:21.984514 20021 raft_consensus.cc:348] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.984602 20021 raft_consensus.cc:374] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:21.984635 20021 raft_consensus.cc:729] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d4b038952284c81a07423e42a349192, State: Initialized, Role: FOLLOWER
I20240501 14:10:21.984763 20021 consensus_queue.cc:260] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.985016 20021 ts_tablet_manager.cc:1398] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:21.985088 20022 ts_tablet_manager.cc:1398] T 2e6457de95c64fb19ae1d294c436c8f7 P 94100faea1434b27a8c6d129d1b562bc: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:10:21.985211 20021 tablet_bootstrap.cc:492] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192: Bootstrap starting.
I20240501 14:10:21.985246 20022 tablet_bootstrap.cc:492] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc: Bootstrap starting.
I20240501 14:10:21.986153 20021 tablet_bootstrap.cc:654] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:21.986536 20023 tablet_bootstrap.cc:492] T ffd3ce2e5a1a4a0182e578d471a022ae P c4d2a3d7c38d4794a1ee78eaef2940d1: No bootstrap required, opened a new log
I20240501 14:10:21.986626 20023 ts_tablet_manager.cc:1367] T ffd3ce2e5a1a4a0182e578d471a022ae P c4d2a3d7c38d4794a1ee78eaef2940d1: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:10:21.987063 20023 raft_consensus.cc:348] T ffd3ce2e5a1a4a0182e578d471a022ae P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.987162 20023 raft_consensus.cc:374] T ffd3ce2e5a1a4a0182e578d471a022ae P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:21.987210 20023 raft_consensus.cc:729] T ffd3ce2e5a1a4a0182e578d471a022ae P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d2a3d7c38d4794a1ee78eaef2940d1, State: Initialized, Role: FOLLOWER
I20240501 14:10:21.987370 20021 tablet_bootstrap.cc:492] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192: No bootstrap required, opened a new log
I20240501 14:10:21.986161 20022 tablet_bootstrap.cc:654] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:21.987545 20021 ts_tablet_manager.cc:1367] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:21.987358 20023 consensus_queue.cc:260] T ffd3ce2e5a1a4a0182e578d471a022ae P c4d2a3d7c38d4794a1ee78eaef2940d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.988013 20021 raft_consensus.cc:348] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.988123 20021 raft_consensus.cc:374] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:21.988157 20021 raft_consensus.cc:729] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d4b038952284c81a07423e42a349192, State: Initialized, Role: FOLLOWER
I20240501 14:10:21.988327 20023 ts_tablet_manager.cc:1398] T ffd3ce2e5a1a4a0182e578d471a022ae P c4d2a3d7c38d4794a1ee78eaef2940d1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:10:21.988287 20021 consensus_queue.cc:260] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.988502 20023 tablet_bootstrap.cc:492] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1: Bootstrap starting.
I20240501 14:10:21.988893 20021 ts_tablet_manager.cc:1398] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:21.989077 20021 tablet_bootstrap.cc:492] T 57d657b87203496ca9d47d00fd98c1af P 7d4b038952284c81a07423e42a349192: Bootstrap starting.
I20240501 14:10:21.989387 20023 tablet_bootstrap.cc:654] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:21.990027 20021 tablet_bootstrap.cc:654] T 57d657b87203496ca9d47d00fd98c1af P 7d4b038952284c81a07423e42a349192: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:21.990231 20022 tablet_bootstrap.cc:492] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc: No bootstrap required, opened a new log
I20240501 14:10:21.990312 20022 ts_tablet_manager.cc:1367] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:10:21.990693 20022 raft_consensus.cc:348] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.990772 20022 raft_consensus.cc:374] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:21.990792 20022 raft_consensus.cc:729] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94100faea1434b27a8c6d129d1b562bc, State: Initialized, Role: FOLLOWER
I20240501 14:10:21.990885 20022 consensus_queue.cc:260] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.991119 20022 ts_tablet_manager.cc:1398] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:21.991326 20022 tablet_bootstrap.cc:492] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc: Bootstrap starting.
I20240501 14:10:21.991679 20021 tablet_bootstrap.cc:492] T 57d657b87203496ca9d47d00fd98c1af P 7d4b038952284c81a07423e42a349192: No bootstrap required, opened a new log
I20240501 14:10:21.991681 20023 tablet_bootstrap.cc:492] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1: No bootstrap required, opened a new log
I20240501 14:10:21.991804 20021 ts_tablet_manager.cc:1367] T 57d657b87203496ca9d47d00fd98c1af P 7d4b038952284c81a07423e42a349192: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:10:21.991816 20023 ts_tablet_manager.cc:1367] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:10:21.992276 20022 tablet_bootstrap.cc:654] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:21.992313 20023 raft_consensus.cc:348] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.992344 20021 raft_consensus.cc:348] T 57d657b87203496ca9d47d00fd98c1af P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.992408 20023 raft_consensus.cc:374] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:21.992470 20021 raft_consensus.cc:374] T 57d657b87203496ca9d47d00fd98c1af P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:21.992476 20023 raft_consensus.cc:729] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d2a3d7c38d4794a1ee78eaef2940d1, State: Initialized, Role: FOLLOWER
I20240501 14:10:21.992525 20021 raft_consensus.cc:729] T 57d657b87203496ca9d47d00fd98c1af P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d4b038952284c81a07423e42a349192, State: Initialized, Role: FOLLOWER
I20240501 14:10:21.992640 20023 consensus_queue.cc:260] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.992671 20021 consensus_queue.cc:260] T 57d657b87203496ca9d47d00fd98c1af P 7d4b038952284c81a07423e42a349192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.992967 20021 ts_tablet_manager.cc:1398] T 57d657b87203496ca9d47d00fd98c1af P 7d4b038952284c81a07423e42a349192: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:21.993146 20021 tablet_bootstrap.cc:492] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192: Bootstrap starting.
I20240501 14:10:21.993194 20023 ts_tablet_manager.cc:1398] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:21.993330 20023 tablet_bootstrap.cc:492] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1: Bootstrap starting.
I20240501 14:10:21.994060 20023 tablet_bootstrap.cc:654] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:21.994755 20021 tablet_bootstrap.cc:654] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:21.994961 20022 tablet_bootstrap.cc:492] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc: No bootstrap required, opened a new log
I20240501 14:10:21.995039 20022 ts_tablet_manager.cc:1367] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:10:21.995436 20022 raft_consensus.cc:348] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.995549 20022 raft_consensus.cc:374] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:21.995604 20022 raft_consensus.cc:729] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94100faea1434b27a8c6d129d1b562bc, State: Initialized, Role: FOLLOWER
I20240501 14:10:21.995729 20022 consensus_queue.cc:260] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.995929 20021 tablet_bootstrap.cc:492] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192: No bootstrap required, opened a new log
I20240501 14:10:21.996022 20021 ts_tablet_manager.cc:1367] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:10:21.996006 20022 ts_tablet_manager.cc:1398] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:21.996369 20022 tablet_bootstrap.cc:492] T 352498cc20094372a957c030cebf375e P 94100faea1434b27a8c6d129d1b562bc: Bootstrap starting.
I20240501 14:10:21.996430 20021 raft_consensus.cc:348] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.996516 20021 raft_consensus.cc:374] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:21.996572 20021 raft_consensus.cc:729] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d4b038952284c81a07423e42a349192, State: Initialized, Role: FOLLOWER
I20240501 14:10:21.996703 20021 consensus_queue.cc:260] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.996963 20021 ts_tablet_manager.cc:1398] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:21.997133 20021 tablet_bootstrap.cc:492] T 9d0888af56f2427c9387c3682f211a58 P 7d4b038952284c81a07423e42a349192: Bootstrap starting.
I20240501 14:10:21.997303 20022 tablet_bootstrap.cc:654] T 352498cc20094372a957c030cebf375e P 94100faea1434b27a8c6d129d1b562bc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:21.998049 20021 tablet_bootstrap.cc:654] T 9d0888af56f2427c9387c3682f211a58 P 7d4b038952284c81a07423e42a349192: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:21.998431 20022 tablet_bootstrap.cc:492] T 352498cc20094372a957c030cebf375e P 94100faea1434b27a8c6d129d1b562bc: No bootstrap required, opened a new log
I20240501 14:10:21.998555 20022 ts_tablet_manager.cc:1367] T 352498cc20094372a957c030cebf375e P 94100faea1434b27a8c6d129d1b562bc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:21.998983 20022 raft_consensus.cc:348] T 352498cc20094372a957c030cebf375e P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.999126 20022 raft_consensus.cc:374] T 352498cc20094372a957c030cebf375e P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:21.999203 20022 raft_consensus.cc:729] T 352498cc20094372a957c030cebf375e P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94100faea1434b27a8c6d129d1b562bc, State: Initialized, Role: FOLLOWER
I20240501 14:10:21.999220 20023 tablet_bootstrap.cc:492] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1: No bootstrap required, opened a new log
I20240501 14:10:21.999408 20023 ts_tablet_manager.cc:1367] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240501 14:10:21.999436 20022 consensus_queue.cc:260] T 352498cc20094372a957c030cebf375e P 94100faea1434b27a8c6d129d1b562bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.999819 20022 ts_tablet_manager.cc:1398] T 352498cc20094372a957c030cebf375e P 94100faea1434b27a8c6d129d1b562bc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:21.999832 20023 raft_consensus.cc:348] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:21.999997 20023 raft_consensus.cc:374] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:22.000041 20023 raft_consensus.cc:729] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d2a3d7c38d4794a1ee78eaef2940d1, State: Initialized, Role: FOLLOWER
I20240501 14:10:22.000157 20022 tablet_bootstrap.cc:492] T a47e39994ef3497e96bedd8a36133e00 P 94100faea1434b27a8c6d129d1b562bc: Bootstrap starting.
I20240501 14:10:22.000140 20023 consensus_queue.cc:260] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.000567 20023 ts_tablet_manager.cc:1398] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:22.000921 20023 tablet_bootstrap.cc:492] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1: Bootstrap starting.
I20240501 14:10:22.001685 20022 tablet_bootstrap.cc:654] T a47e39994ef3497e96bedd8a36133e00 P 94100faea1434b27a8c6d129d1b562bc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:22.001896 20023 tablet_bootstrap.cc:654] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:22.002399 20021 tablet_bootstrap.cc:492] T 9d0888af56f2427c9387c3682f211a58 P 7d4b038952284c81a07423e42a349192: No bootstrap required, opened a new log
I20240501 14:10:22.002478 20021 ts_tablet_manager.cc:1367] T 9d0888af56f2427c9387c3682f211a58 P 7d4b038952284c81a07423e42a349192: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:10:22.002878 20021 raft_consensus.cc:348] T 9d0888af56f2427c9387c3682f211a58 P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.002974 20021 raft_consensus.cc:374] T 9d0888af56f2427c9387c3682f211a58 P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:22.002990 20022 tablet_bootstrap.cc:492] T a47e39994ef3497e96bedd8a36133e00 P 94100faea1434b27a8c6d129d1b562bc: No bootstrap required, opened a new log
I20240501 14:10:22.003010 20021 raft_consensus.cc:729] T 9d0888af56f2427c9387c3682f211a58 P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d4b038952284c81a07423e42a349192, State: Initialized, Role: FOLLOWER
I20240501 14:10:22.003096 20022 ts_tablet_manager.cc:1367] T a47e39994ef3497e96bedd8a36133e00 P 94100faea1434b27a8c6d129d1b562bc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:10:22.003203 20023 tablet_bootstrap.cc:492] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1: No bootstrap required, opened a new log
I20240501 14:10:22.003296 20023 ts_tablet_manager.cc:1367] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:10:22.003206 20021 consensus_queue.cc:260] T 9d0888af56f2427c9387c3682f211a58 P 7d4b038952284c81a07423e42a349192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.003556 20022 raft_consensus.cc:348] T a47e39994ef3497e96bedd8a36133e00 P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.003659 20022 raft_consensus.cc:374] T a47e39994ef3497e96bedd8a36133e00 P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:22.003706 20022 raft_consensus.cc:729] T a47e39994ef3497e96bedd8a36133e00 P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94100faea1434b27a8c6d129d1b562bc, State: Initialized, Role: FOLLOWER
I20240501 14:10:22.003690 20023 raft_consensus.cc:348] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.003770 20023 raft_consensus.cc:374] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:22.003801 20023 raft_consensus.cc:729] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d2a3d7c38d4794a1ee78eaef2940d1, State: Initialized, Role: FOLLOWER
I20240501 14:10:22.003824 20022 consensus_queue.cc:260] T a47e39994ef3497e96bedd8a36133e00 P 94100faea1434b27a8c6d129d1b562bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.003959 20023 consensus_queue.cc:260] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.004129 20022 ts_tablet_manager.cc:1398] T a47e39994ef3497e96bedd8a36133e00 P 94100faea1434b27a8c6d129d1b562bc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:22.004297 20022 tablet_bootstrap.cc:492] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc: Bootstrap starting.
I20240501 14:10:22.004519 20021 ts_tablet_manager.cc:1398] T 9d0888af56f2427c9387c3682f211a58 P 7d4b038952284c81a07423e42a349192: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:10:22.004671 20021 tablet_bootstrap.cc:492] T 4b28da769e1f46e1b05bd25e8428316f P 7d4b038952284c81a07423e42a349192: Bootstrap starting.
I20240501 14:10:22.004786 20023 ts_tablet_manager.cc:1398] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:22.004958 20023 tablet_bootstrap.cc:492] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1: Bootstrap starting.
I20240501 14:10:22.005213 20022 tablet_bootstrap.cc:654] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:22.005558 20021 tablet_bootstrap.cc:654] T 4b28da769e1f46e1b05bd25e8428316f P 7d4b038952284c81a07423e42a349192: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:22.005952 20023 tablet_bootstrap.cc:654] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:22.006722 20021 tablet_bootstrap.cc:492] T 4b28da769e1f46e1b05bd25e8428316f P 7d4b038952284c81a07423e42a349192: No bootstrap required, opened a new log
I20240501 14:10:22.006815 20021 ts_tablet_manager.cc:1367] T 4b28da769e1f46e1b05bd25e8428316f P 7d4b038952284c81a07423e42a349192: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:22.007057 20022 tablet_bootstrap.cc:492] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc: No bootstrap required, opened a new log
I20240501 14:10:22.007139 20022 ts_tablet_manager.cc:1367] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:10:22.007234 20021 raft_consensus.cc:348] T 4b28da769e1f46e1b05bd25e8428316f P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.007342 20021 raft_consensus.cc:374] T 4b28da769e1f46e1b05bd25e8428316f P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:22.007375 20021 raft_consensus.cc:729] T 4b28da769e1f46e1b05bd25e8428316f P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d4b038952284c81a07423e42a349192, State: Initialized, Role: FOLLOWER
I20240501 14:10:22.007485 20021 consensus_queue.cc:260] T 4b28da769e1f46e1b05bd25e8428316f P 7d4b038952284c81a07423e42a349192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.007552 20022 raft_consensus.cc:348] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.007633 20022 raft_consensus.cc:374] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:22.007671 20022 raft_consensus.cc:729] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94100faea1434b27a8c6d129d1b562bc, State: Initialized, Role: FOLLOWER
I20240501 14:10:22.007781 20021 ts_tablet_manager.cc:1398] T 4b28da769e1f46e1b05bd25e8428316f P 7d4b038952284c81a07423e42a349192: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:22.007790 20022 consensus_queue.cc:260] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.007980 20021 tablet_bootstrap.cc:492] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192: Bootstrap starting.
I20240501 14:10:22.008173 20022 ts_tablet_manager.cc:1398] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:22.008358 20022 tablet_bootstrap.cc:492] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc: Bootstrap starting.
I20240501 14:10:22.008884 20021 tablet_bootstrap.cc:654] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:22.009294 20022 tablet_bootstrap.cc:654] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:22.010804 20021 tablet_bootstrap.cc:492] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192: No bootstrap required, opened a new log
I20240501 14:10:22.010882 20022 tablet_bootstrap.cc:492] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc: No bootstrap required, opened a new log
I20240501 14:10:22.010900 20021 ts_tablet_manager.cc:1367] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:10:22.010995 20022 ts_tablet_manager.cc:1367] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:10:22.011428 20022 raft_consensus.cc:348] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.011505 20022 raft_consensus.cc:374] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:22.011525 20022 raft_consensus.cc:729] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94100faea1434b27a8c6d129d1b562bc, State: Initialized, Role: FOLLOWER
I20240501 14:10:22.011643 20023 tablet_bootstrap.cc:492] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1: No bootstrap required, opened a new log
I20240501 14:10:22.011617 20022 consensus_queue.cc:260] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.011627 20021 raft_consensus.cc:348] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.011797 20023 ts_tablet_manager.cc:1367] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:10:22.011822 20021 raft_consensus.cc:374] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:22.011871 20021 raft_consensus.cc:729] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d4b038952284c81a07423e42a349192, State: Initialized, Role: FOLLOWER
I20240501 14:10:22.012008 20022 ts_tablet_manager.cc:1398] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:22.012174 20022 tablet_bootstrap.cc:492] T 4b28da769e1f46e1b05bd25e8428316f P 94100faea1434b27a8c6d129d1b562bc: Bootstrap starting.
I20240501 14:10:22.012223 20023 raft_consensus.cc:348] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.012323 20023 raft_consensus.cc:374] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:22.012359 20023 raft_consensus.cc:729] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d2a3d7c38d4794a1ee78eaef2940d1, State: Initialized, Role: FOLLOWER
I20240501 14:10:22.012483 20023 consensus_queue.cc:260] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.012055 20021 consensus_queue.cc:260] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.012748 20023 ts_tablet_manager.cc:1398] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:22.012890 20023 tablet_bootstrap.cc:492] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1: Bootstrap starting.
I20240501 14:10:22.013130 20022 tablet_bootstrap.cc:654] T 4b28da769e1f46e1b05bd25e8428316f P 94100faea1434b27a8c6d129d1b562bc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:22.013286 20021 ts_tablet_manager.cc:1398] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:10:22.013767 20023 tablet_bootstrap.cc:654] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:22.014257 20022 tablet_bootstrap.cc:492] T 4b28da769e1f46e1b05bd25e8428316f P 94100faea1434b27a8c6d129d1b562bc: No bootstrap required, opened a new log
I20240501 14:10:22.014353 20022 ts_tablet_manager.cc:1367] T 4b28da769e1f46e1b05bd25e8428316f P 94100faea1434b27a8c6d129d1b562bc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:22.014767 20022 raft_consensus.cc:348] T 4b28da769e1f46e1b05bd25e8428316f P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.014873 20022 raft_consensus.cc:374] T 4b28da769e1f46e1b05bd25e8428316f P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:22.014904 20022 raft_consensus.cc:729] T 4b28da769e1f46e1b05bd25e8428316f P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94100faea1434b27a8c6d129d1b562bc, State: Initialized, Role: FOLLOWER
I20240501 14:10:22.015044 20022 consensus_queue.cc:260] T 4b28da769e1f46e1b05bd25e8428316f P 94100faea1434b27a8c6d129d1b562bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.015385 20022 ts_tablet_manager.cc:1398] T 4b28da769e1f46e1b05bd25e8428316f P 94100faea1434b27a8c6d129d1b562bc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:22.015448 20023 tablet_bootstrap.cc:492] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1: No bootstrap required, opened a new log
I20240501 14:10:22.015511 20023 ts_tablet_manager.cc:1367] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:10:22.015790 20023 raft_consensus.cc:348] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.015875 20023 raft_consensus.cc:374] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:22.015894 20023 raft_consensus.cc:729] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d2a3d7c38d4794a1ee78eaef2940d1, State: Initialized, Role: FOLLOWER
I20240501 14:10:22.016026 20023 consensus_queue.cc:260] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.016219 20023 ts_tablet_manager.cc:1398] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:22.016335 20023 tablet_bootstrap.cc:492] T 2e6457de95c64fb19ae1d294c436c8f7 P c4d2a3d7c38d4794a1ee78eaef2940d1: Bootstrap starting.
I20240501 14:10:22.017190 20023 tablet_bootstrap.cc:654] T 2e6457de95c64fb19ae1d294c436c8f7 P c4d2a3d7c38d4794a1ee78eaef2940d1: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:22.018157 20023 tablet_bootstrap.cc:492] T 2e6457de95c64fb19ae1d294c436c8f7 P c4d2a3d7c38d4794a1ee78eaef2940d1: No bootstrap required, opened a new log
I20240501 14:10:22.018240 20023 ts_tablet_manager.cc:1367] T 2e6457de95c64fb19ae1d294c436c8f7 P c4d2a3d7c38d4794a1ee78eaef2940d1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:22.018573 20023 raft_consensus.cc:348] T 2e6457de95c64fb19ae1d294c436c8f7 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.018658 20023 raft_consensus.cc:374] T 2e6457de95c64fb19ae1d294c436c8f7 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:22.018692 20023 raft_consensus.cc:729] T 2e6457de95c64fb19ae1d294c436c8f7 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d2a3d7c38d4794a1ee78eaef2940d1, State: Initialized, Role: FOLLOWER
I20240501 14:10:22.018811 20023 consensus_queue.cc:260] T 2e6457de95c64fb19ae1d294c436c8f7 P c4d2a3d7c38d4794a1ee78eaef2940d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.019028 20023 ts_tablet_manager.cc:1398] T 2e6457de95c64fb19ae1d294c436c8f7 P c4d2a3d7c38d4794a1ee78eaef2940d1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:22.019188 20023 tablet_bootstrap.cc:492] T 352498cc20094372a957c030cebf375e P c4d2a3d7c38d4794a1ee78eaef2940d1: Bootstrap starting.
I20240501 14:10:22.019868 20023 tablet_bootstrap.cc:654] T 352498cc20094372a957c030cebf375e P c4d2a3d7c38d4794a1ee78eaef2940d1: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:22.020862 20023 tablet_bootstrap.cc:492] T 352498cc20094372a957c030cebf375e P c4d2a3d7c38d4794a1ee78eaef2940d1: No bootstrap required, opened a new log
I20240501 14:10:22.020943 20023 ts_tablet_manager.cc:1367] T 352498cc20094372a957c030cebf375e P c4d2a3d7c38d4794a1ee78eaef2940d1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:22.021723 20023 raft_consensus.cc:348] T 352498cc20094372a957c030cebf375e P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.021840 20023 raft_consensus.cc:374] T 352498cc20094372a957c030cebf375e P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:22.021875 20023 raft_consensus.cc:729] T 352498cc20094372a957c030cebf375e P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d2a3d7c38d4794a1ee78eaef2940d1, State: Initialized, Role: FOLLOWER
I20240501 14:10:22.022011 20023 consensus_queue.cc:260] T 352498cc20094372a957c030cebf375e P c4d2a3d7c38d4794a1ee78eaef2940d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.022266 20023 ts_tablet_manager.cc:1398] T 352498cc20094372a957c030cebf375e P c4d2a3d7c38d4794a1ee78eaef2940d1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:22.029867 20027 raft_consensus.cc:482] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:22.029978 20027 raft_consensus.cc:504] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.030489 20027 leader_election.cc:288] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591), 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:22.034094 19975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f796e2ca050e4666b9eb2dfd8a9ee2ff" candidate_uuid: "7d4b038952284c81a07423e42a349192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94100faea1434b27a8c6d129d1b562bc" is_pre_election: true
I20240501 14:10:22.034096 19900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f796e2ca050e4666b9eb2dfd8a9ee2ff" candidate_uuid: "7d4b038952284c81a07423e42a349192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" is_pre_election: true
I20240501 14:10:22.034248 19975 raft_consensus.cc:2454] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d4b038952284c81a07423e42a349192 in term 0.
I20240501 14:10:22.034322 19900 raft_consensus.cc:2454] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d4b038952284c81a07423e42a349192 in term 0.
I20240501 14:10:22.034574 19790 leader_election.cc:302] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4b038952284c81a07423e42a349192, 94100faea1434b27a8c6d129d1b562bc; no voters:
I20240501 14:10:22.034745 20027 raft_consensus.cc:2781] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:22.034818 20027 raft_consensus.cc:482] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:22.034875 20027 raft_consensus.cc:3037] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.035776 20027 raft_consensus.cc:504] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.036151 20027 leader_election.cc:288] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [CANDIDATE]: Term 1 election: Requested vote from peers c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591), 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:22.036484 19900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f796e2ca050e4666b9eb2dfd8a9ee2ff" candidate_uuid: "7d4b038952284c81a07423e42a349192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
I20240501 14:10:22.036621 19900 raft_consensus.cc:3037] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.036605 19975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f796e2ca050e4666b9eb2dfd8a9ee2ff" candidate_uuid: "7d4b038952284c81a07423e42a349192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
I20240501 14:10:22.036767 19975 raft_consensus.cc:3037] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.037477 19900 raft_consensus.cc:2454] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d4b038952284c81a07423e42a349192 in term 1.
I20240501 14:10:22.037542 19975 raft_consensus.cc:2454] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d4b038952284c81a07423e42a349192 in term 1.
I20240501 14:10:22.037815 19788 leader_election.cc:302] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7d4b038952284c81a07423e42a349192, 94100faea1434b27a8c6d129d1b562bc, c4d2a3d7c38d4794a1ee78eaef2940d1; no voters:
I20240501 14:10:22.037983 20027 raft_consensus.cc:2781] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:22.038179 20027 raft_consensus.cc:686] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 1 LEADER]: Becoming Leader. State: Replica: 7d4b038952284c81a07423e42a349192, State: Running, Role: LEADER
I20240501 14:10:22.038311 20027 consensus_queue.cc:237] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.039904 19719 catalog_manager.cc:5484] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d4b038952284c81a07423e42a349192 (127.16.13.193). New cstate: current_term: 1 leader_uuid: "7d4b038952284c81a07423e42a349192" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:22.045851 20025 raft_consensus.cc:482] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:22.045958 20025 raft_consensus.cc:504] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.046411 20025 leader_election.cc:288] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811), 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:22.049288 20032 raft_consensus.cc:482] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:22.049432 20032 raft_consensus.cc:504] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.049932 20032 leader_election.cc:288] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811), c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591)
I20240501 14:10:22.050606 19824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8922efb08b3491fa8d947793206bf7a" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4b038952284c81a07423e42a349192" is_pre_election: true
I20240501 14:10:22.050735 19824 raft_consensus.cc:2454] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 0.
I20240501 14:10:22.051084 19866 leader_election.cc:302] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4b038952284c81a07423e42a349192, c4d2a3d7c38d4794a1ee78eaef2940d1; no voters:
I20240501 14:10:22.051146 19975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8922efb08b3491fa8d947793206bf7a" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94100faea1434b27a8c6d129d1b562bc" is_pre_election: true
I20240501 14:10:22.051249 19975 raft_consensus.cc:2454] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 0.
I20240501 14:10:22.051254 20025 raft_consensus.cc:2781] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:22.051352 20025 raft_consensus.cc:482] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:22.051400 20025 raft_consensus.cc:3037] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.052707 20025 raft_consensus.cc:504] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.053025 20025 leader_election.cc:288] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 1 election: Requested vote from peers 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811), 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:22.053371 19975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8922efb08b3491fa8d947793206bf7a" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
I20240501 14:10:22.053484 19975 raft_consensus.cc:3037] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.053519 19824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8922efb08b3491fa8d947793206bf7a" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4b038952284c81a07423e42a349192"
I20240501 14:10:22.053614 19824 raft_consensus.cc:3037] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.054224 19975 raft_consensus.cc:2454] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 1.
I20240501 14:10:22.054805 19824 raft_consensus.cc:2454] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 1.
I20240501 14:10:22.055061 19866 leader_election.cc:302] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4b038952284c81a07423e42a349192, c4d2a3d7c38d4794a1ee78eaef2940d1; no voters:
I20240501 14:10:22.055207 20025 raft_consensus.cc:2781] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:22.055233 19900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8922efb08b3491fa8d947793206bf7a" candidate_uuid: "94100faea1434b27a8c6d129d1b562bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" is_pre_election: true
I20240501 14:10:22.055392 20025 raft_consensus.cc:686] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Becoming Leader. State: Replica: c4d2a3d7c38d4794a1ee78eaef2940d1, State: Running, Role: LEADER
I20240501 14:10:22.055542 20025 consensus_queue.cc:237] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.055845 19824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8922efb08b3491fa8d947793206bf7a" candidate_uuid: "94100faea1434b27a8c6d129d1b562bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4b038952284c81a07423e42a349192" is_pre_election: true
I20240501 14:10:22.056002 19824 raft_consensus.cc:2379] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 94100faea1434b27a8c6d129d1b562bc in current term 1: Already voted for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in this term.
I20240501 14:10:22.056761 19939 leader_election.cc:302] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94100faea1434b27a8c6d129d1b562bc; no voters: 7d4b038952284c81a07423e42a349192, c4d2a3d7c38d4794a1ee78eaef2940d1
I20240501 14:10:22.056943 20032 raft_consensus.cc:2726] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:10:22.057607 19719 catalog_manager.cc:5484] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194). New cstate: current_term: 1 leader_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:22.059482 20027 raft_consensus.cc:482] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:22.059577 20027 raft_consensus.cc:504] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.059850 20027 leader_election.cc:288] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591), 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:22.060173 19975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47e39994ef3497e96bedd8a36133e00" candidate_uuid: "7d4b038952284c81a07423e42a349192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94100faea1434b27a8c6d129d1b562bc" is_pre_election: true
I20240501 14:10:22.060148 19900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47e39994ef3497e96bedd8a36133e00" candidate_uuid: "7d4b038952284c81a07423e42a349192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" is_pre_election: true
I20240501 14:10:22.060271 19975 raft_consensus.cc:2454] T a47e39994ef3497e96bedd8a36133e00 P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d4b038952284c81a07423e42a349192 in term 0.
I20240501 14:10:22.060271 19900 raft_consensus.cc:2454] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d4b038952284c81a07423e42a349192 in term 0.
I20240501 14:10:22.060596 19788 leader_election.cc:302] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4b038952284c81a07423e42a349192, c4d2a3d7c38d4794a1ee78eaef2940d1; no voters:
I20240501 14:10:22.060757 20027 raft_consensus.cc:2781] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:22.060817 20027 raft_consensus.cc:482] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:22.060870 20027 raft_consensus.cc:3037] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.061779 20027 raft_consensus.cc:504] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.062100 20027 leader_election.cc:288] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [CANDIDATE]: Term 1 election: Requested vote from peers c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591), 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:22.062404 19900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47e39994ef3497e96bedd8a36133e00" candidate_uuid: "7d4b038952284c81a07423e42a349192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
I20240501 14:10:22.062444 19975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47e39994ef3497e96bedd8a36133e00" candidate_uuid: "7d4b038952284c81a07423e42a349192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
I20240501 14:10:22.062512 19900 raft_consensus.cc:3037] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.062527 19975 raft_consensus.cc:3037] T a47e39994ef3497e96bedd8a36133e00 P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.063225 19975 raft_consensus.cc:2454] T a47e39994ef3497e96bedd8a36133e00 P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d4b038952284c81a07423e42a349192 in term 1.
I20240501 14:10:22.063232 19900 raft_consensus.cc:2454] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d4b038952284c81a07423e42a349192 in term 1.
I20240501 14:10:22.063477 19790 leader_election.cc:302] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4b038952284c81a07423e42a349192, 94100faea1434b27a8c6d129d1b562bc; no voters:
I20240501 14:10:22.063685 20027 raft_consensus.cc:2781] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:22.063764 20027 raft_consensus.cc:686] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [term 1 LEADER]: Becoming Leader. State: Replica: 7d4b038952284c81a07423e42a349192, State: Running, Role: LEADER
I20240501 14:10:22.063889 20027 consensus_queue.cc:237] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.065627 19719 catalog_manager.cc:5484] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d4b038952284c81a07423e42a349192 (127.16.13.193). New cstate: current_term: 1 leader_uuid: "7d4b038952284c81a07423e42a349192" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:22.066723 20025 raft_consensus.cc:482] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:22.066813 20025 raft_consensus.cc:504] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.067087 20025 leader_election.cc:288] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811), 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:22.067394 19824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515893894ce0478caa58378e6c2e047b" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4b038952284c81a07423e42a349192" is_pre_election: true
I20240501 14:10:22.067466 19975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515893894ce0478caa58378e6c2e047b" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94100faea1434b27a8c6d129d1b562bc" is_pre_election: true
I20240501 14:10:22.067504 19824 raft_consensus.cc:2454] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 0.
I20240501 14:10:22.067595 19975 raft_consensus.cc:2454] T 515893894ce0478caa58378e6c2e047b P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 0.
I20240501 14:10:22.067838 19866 leader_election.cc:302] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4b038952284c81a07423e42a349192, c4d2a3d7c38d4794a1ee78eaef2940d1; no voters:
I20240501 14:10:22.068048 20025 raft_consensus.cc:2781] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:22.068109 20025 raft_consensus.cc:482] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:22.068149 20025 raft_consensus.cc:3037] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.068883 20025 raft_consensus.cc:504] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.069166 20025 leader_election.cc:288] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 1 election: Requested vote from peers 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811), 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:22.069489 19824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515893894ce0478caa58378e6c2e047b" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4b038952284c81a07423e42a349192"
I20240501 14:10:22.069489 19975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515893894ce0478caa58378e6c2e047b" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
I20240501 14:10:22.069635 19975 raft_consensus.cc:3037] T 515893894ce0478caa58378e6c2e047b P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.069635 19824 raft_consensus.cc:3037] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.070523 19975 raft_consensus.cc:2454] T 515893894ce0478caa58378e6c2e047b P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 1.
I20240501 14:10:22.070523 19824 raft_consensus.cc:2454] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 1.
I20240501 14:10:22.070819 19866 leader_election.cc:302] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94100faea1434b27a8c6d129d1b562bc, c4d2a3d7c38d4794a1ee78eaef2940d1; no voters:
I20240501 14:10:22.070961 20025 raft_consensus.cc:2781] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:22.071048 20025 raft_consensus.cc:686] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Becoming Leader. State: Replica: c4d2a3d7c38d4794a1ee78eaef2940d1, State: Running, Role: LEADER
I20240501 14:10:22.071192 20025 consensus_queue.cc:237] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.072741 19719 catalog_manager.cc:5484] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194). New cstate: current_term: 1 leader_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:22.085024 20025 raft_consensus.cc:482] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:22.085155 20025 raft_consensus.cc:504] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.085551 20025 leader_election.cc:288] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811), 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:22.085919 19975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d657b87203496ca9d47d00fd98c1af" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94100faea1434b27a8c6d129d1b562bc" is_pre_election: true
I20240501 14:10:22.085919 19824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d657b87203496ca9d47d00fd98c1af" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4b038952284c81a07423e42a349192" is_pre_election: true
I20240501 14:10:22.086076 19975 raft_consensus.cc:2454] T 57d657b87203496ca9d47d00fd98c1af P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 0.
I20240501 14:10:22.086088 19824 raft_consensus.cc:2454] T 57d657b87203496ca9d47d00fd98c1af P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 0.
I20240501 14:10:22.086530 19866 leader_election.cc:302] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94100faea1434b27a8c6d129d1b562bc, c4d2a3d7c38d4794a1ee78eaef2940d1; no voters:
I20240501 14:10:22.086714 20025 raft_consensus.cc:2781] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:22.086779 20025 raft_consensus.cc:482] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:22.086833 20025 raft_consensus.cc:3037] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.087695 20025 raft_consensus.cc:504] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.088116 20025 leader_election.cc:288] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 1 election: Requested vote from peers 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811), 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:22.088454 19824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d657b87203496ca9d47d00fd98c1af" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4b038952284c81a07423e42a349192"
I20240501 14:10:22.088474 19975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d657b87203496ca9d47d00fd98c1af" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
I20240501 14:10:22.088552 19824 raft_consensus.cc:3037] T 57d657b87203496ca9d47d00fd98c1af P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.088577 19975 raft_consensus.cc:3037] T 57d657b87203496ca9d47d00fd98c1af P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.089336 19824 raft_consensus.cc:2454] T 57d657b87203496ca9d47d00fd98c1af P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 1.
I20240501 14:10:22.089344 19975 raft_consensus.cc:2454] T 57d657b87203496ca9d47d00fd98c1af P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 1.
I20240501 14:10:22.089624 19866 leader_election.cc:302] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4b038952284c81a07423e42a349192, c4d2a3d7c38d4794a1ee78eaef2940d1; no voters:
I20240501 14:10:22.089808 20025 raft_consensus.cc:2781] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:22.089881 20025 raft_consensus.cc:686] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Becoming Leader. State: Replica: c4d2a3d7c38d4794a1ee78eaef2940d1, State: Running, Role: LEADER
I20240501 14:10:22.090016 20025 consensus_queue.cc:237] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.091641 19719 catalog_manager.cc:5484] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194). New cstate: current_term: 1 leader_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:22.119098 20027 raft_consensus.cc:482] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:22.119287 20027 raft_consensus.cc:504] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.119737 20027 leader_election.cc:288] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591), 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:22.120052 19975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "352498cc20094372a957c030cebf375e" candidate_uuid: "7d4b038952284c81a07423e42a349192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94100faea1434b27a8c6d129d1b562bc" is_pre_election: true
I20240501 14:10:22.120083 19900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "352498cc20094372a957c030cebf375e" candidate_uuid: "7d4b038952284c81a07423e42a349192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" is_pre_election: true
I20240501 14:10:22.120186 19975 raft_consensus.cc:2454] T 352498cc20094372a957c030cebf375e P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d4b038952284c81a07423e42a349192 in term 0.
I20240501 14:10:22.120209 19900 raft_consensus.cc:2454] T 352498cc20094372a957c030cebf375e P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d4b038952284c81a07423e42a349192 in term 0.
I20240501 14:10:22.120455 19788 leader_election.cc:302] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4b038952284c81a07423e42a349192, c4d2a3d7c38d4794a1ee78eaef2940d1; no voters:
I20240501 14:10:22.120615 20027 raft_consensus.cc:2781] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:22.120671 20027 raft_consensus.cc:482] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:22.120714 20027 raft_consensus.cc:3037] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.121621 20027 raft_consensus.cc:504] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.121913 20027 leader_election.cc:288] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [CANDIDATE]: Term 1 election: Requested vote from peers c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591), 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:22.122412 19900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "352498cc20094372a957c030cebf375e" candidate_uuid: "7d4b038952284c81a07423e42a349192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
I20240501 14:10:22.122414 19975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "352498cc20094372a957c030cebf375e" candidate_uuid: "7d4b038952284c81a07423e42a349192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
I20240501 14:10:22.122612 19900 raft_consensus.cc:3037] T 352498cc20094372a957c030cebf375e P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.122654 19975 raft_consensus.cc:3037] T 352498cc20094372a957c030cebf375e P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.123375 19900 raft_consensus.cc:2454] T 352498cc20094372a957c030cebf375e P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d4b038952284c81a07423e42a349192 in term 1.
I20240501 14:10:22.123414 20032 raft_consensus.cc:482] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:22.123414 19975 raft_consensus.cc:2454] T 352498cc20094372a957c030cebf375e P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d4b038952284c81a07423e42a349192 in term 1.
I20240501 14:10:22.123525 20032 raft_consensus.cc:504] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.123675 19788 leader_election.cc:302] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4b038952284c81a07423e42a349192, c4d2a3d7c38d4794a1ee78eaef2940d1; no voters:
I20240501 14:10:22.123811 20027 raft_consensus.cc:2781] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:22.123900 20032 leader_election.cc:288] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811), c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591)
I20240501 14:10:22.124142 20027 raft_consensus.cc:686] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [term 1 LEADER]: Becoming Leader. State: Replica: 7d4b038952284c81a07423e42a349192, State: Running, Role: LEADER
I20240501 14:10:22.124320 20027 consensus_queue.cc:237] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.124408 19824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd3ce2e5a1a4a0182e578d471a022ae" candidate_uuid: "94100faea1434b27a8c6d129d1b562bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4b038952284c81a07423e42a349192" is_pre_election: true
I20240501 14:10:22.124526 19824 raft_consensus.cc:2454] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94100faea1434b27a8c6d129d1b562bc in term 0.
I20240501 14:10:22.124621 19900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd3ce2e5a1a4a0182e578d471a022ae" candidate_uuid: "94100faea1434b27a8c6d129d1b562bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" is_pre_election: true
I20240501 14:10:22.124720 19900 raft_consensus.cc:2454] T ffd3ce2e5a1a4a0182e578d471a022ae P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94100faea1434b27a8c6d129d1b562bc in term 0.
I20240501 14:10:22.124764 19941 leader_election.cc:302] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4b038952284c81a07423e42a349192, 94100faea1434b27a8c6d129d1b562bc; no voters:
I20240501 14:10:22.124917 20032 raft_consensus.cc:2781] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:22.124974 20032 raft_consensus.cc:482] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:22.125016 20032 raft_consensus.cc:3037] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.125988 20032 raft_consensus.cc:504] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.126224 19719 catalog_manager.cc:5484] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d4b038952284c81a07423e42a349192 (127.16.13.193). New cstate: current_term: 1 leader_uuid: "7d4b038952284c81a07423e42a349192" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:22.126340 20032 leader_election.cc:288] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [CANDIDATE]: Term 1 election: Requested vote from peers 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811), c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591)
I20240501 14:10:22.126610 19824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd3ce2e5a1a4a0182e578d471a022ae" candidate_uuid: "94100faea1434b27a8c6d129d1b562bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4b038952284c81a07423e42a349192"
I20240501 14:10:22.126623 19900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd3ce2e5a1a4a0182e578d471a022ae" candidate_uuid: "94100faea1434b27a8c6d129d1b562bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
I20240501 14:10:22.126746 19824 raft_consensus.cc:3037] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.126772 19900 raft_consensus.cc:3037] T ffd3ce2e5a1a4a0182e578d471a022ae P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.127568 19824 raft_consensus.cc:2454] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94100faea1434b27a8c6d129d1b562bc in term 1.
I20240501 14:10:22.127568 19900 raft_consensus.cc:2454] T ffd3ce2e5a1a4a0182e578d471a022ae P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94100faea1434b27a8c6d129d1b562bc in term 1.
I20240501 14:10:22.127897 19939 leader_election.cc:302] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94100faea1434b27a8c6d129d1b562bc, c4d2a3d7c38d4794a1ee78eaef2940d1; no voters:
I20240501 14:10:22.128075 20032 raft_consensus.cc:2781] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:22.128177 20032 raft_consensus.cc:686] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [term 1 LEADER]: Becoming Leader. State: Replica: 94100faea1434b27a8c6d129d1b562bc, State: Running, Role: LEADER
I20240501 14:10:22.128285 20032 consensus_queue.cc:237] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.129904 19719 catalog_manager.cc:5484] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 94100faea1434b27a8c6d129d1b562bc (127.16.13.195). New cstate: current_term: 1 leader_uuid: "94100faea1434b27a8c6d129d1b562bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: HEALTHY } } }
I20240501 14:10:22.176576 20025 raft_consensus.cc:482] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:22.176735 20025 raft_consensus.cc:504] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.177145 20025 leader_election.cc:288] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811), 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:22.177699 19824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b28da769e1f46e1b05bd25e8428316f" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4b038952284c81a07423e42a349192" is_pre_election: true
I20240501 14:10:22.177717 19975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b28da769e1f46e1b05bd25e8428316f" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94100faea1434b27a8c6d129d1b562bc" is_pre_election: true
I20240501 14:10:22.177824 19824 raft_consensus.cc:2454] T 4b28da769e1f46e1b05bd25e8428316f P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 0.
I20240501 14:10:22.177894 19975 raft_consensus.cc:2454] T 4b28da769e1f46e1b05bd25e8428316f P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 0.
I20240501 14:10:22.178148 19866 leader_election.cc:302] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4b038952284c81a07423e42a349192, c4d2a3d7c38d4794a1ee78eaef2940d1; no voters:
I20240501 14:10:22.178320 20025 raft_consensus.cc:2781] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:22.178382 20025 raft_consensus.cc:482] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:22.178439 20025 raft_consensus.cc:3037] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.179338 20025 raft_consensus.cc:504] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.179689 20025 leader_election.cc:288] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 1 election: Requested vote from peers 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811), 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:22.180099 19824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b28da769e1f46e1b05bd25e8428316f" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4b038952284c81a07423e42a349192"
I20240501 14:10:22.180191 19975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b28da769e1f46e1b05bd25e8428316f" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
I20240501 14:10:22.180218 19824 raft_consensus.cc:3037] T 4b28da769e1f46e1b05bd25e8428316f P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.180322 19975 raft_consensus.cc:3037] T 4b28da769e1f46e1b05bd25e8428316f P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.181000 19975 raft_consensus.cc:2454] T 4b28da769e1f46e1b05bd25e8428316f P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 1.
I20240501 14:10:22.181001 19824 raft_consensus.cc:2454] T 4b28da769e1f46e1b05bd25e8428316f P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 1.
I20240501 14:10:22.181241 19866 leader_election.cc:302] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94100faea1434b27a8c6d129d1b562bc, c4d2a3d7c38d4794a1ee78eaef2940d1; no voters:
I20240501 14:10:22.181480 20025 raft_consensus.cc:2781] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:22.181566 20025 raft_consensus.cc:686] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Becoming Leader. State: Replica: c4d2a3d7c38d4794a1ee78eaef2940d1, State: Running, Role: LEADER
I20240501 14:10:22.181861 20025 consensus_queue.cc:237] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.183625 19719 catalog_manager.cc:5484] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194). New cstate: current_term: 1 leader_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:22.188963 20027 raft_consensus.cc:482] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:22.189059 20027 raft_consensus.cc:504] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.189432 20027 leader_election.cc:288] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591), 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:22.189569 20025 raft_consensus.cc:482] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:22.189652 20025 raft_consensus.cc:504] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.189842 19975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e6457de95c64fb19ae1d294c436c8f7" candidate_uuid: "7d4b038952284c81a07423e42a349192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94100faea1434b27a8c6d129d1b562bc" is_pre_election: true
I20240501 14:10:22.189842 19900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e6457de95c64fb19ae1d294c436c8f7" candidate_uuid: "7d4b038952284c81a07423e42a349192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" is_pre_election: true
I20240501 14:10:22.189965 19975 raft_consensus.cc:2454] T 2e6457de95c64fb19ae1d294c436c8f7 P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d4b038952284c81a07423e42a349192 in term 0.
I20240501 14:10:22.190013 19900 raft_consensus.cc:2454] T 2e6457de95c64fb19ae1d294c436c8f7 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d4b038952284c81a07423e42a349192 in term 0.
I20240501 14:10:22.190251 19790 leader_election.cc:302] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4b038952284c81a07423e42a349192, 94100faea1434b27a8c6d129d1b562bc; no voters:
I20240501 14:10:22.190321 19975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0888af56f2427c9387c3682f211a58" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94100faea1434b27a8c6d129d1b562bc" is_pre_election: true
I20240501 14:10:22.190409 20027 raft_consensus.cc:2781] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:22.190416 19975 raft_consensus.cc:2454] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 0.
I20240501 14:10:22.189963 20025 leader_election.cc:288] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811), 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:22.190423 19824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0888af56f2427c9387c3682f211a58" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4b038952284c81a07423e42a349192" is_pre_election: true
I20240501 14:10:22.190477 20027 raft_consensus.cc:482] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:22.190620 20027 raft_consensus.cc:3037] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.190626 19824 raft_consensus.cc:2454] T 9d0888af56f2427c9387c3682f211a58 P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 0.
I20240501 14:10:22.190724 19866 leader_election.cc:302] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94100faea1434b27a8c6d129d1b562bc, c4d2a3d7c38d4794a1ee78eaef2940d1; no voters:
I20240501 14:10:22.190976 20025 raft_consensus.cc:2781] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:22.191044 20025 raft_consensus.cc:482] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:22.191087 20025 raft_consensus.cc:3037] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.191529 20027 raft_consensus.cc:504] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.191762 20025 raft_consensus.cc:504] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.191850 20027 leader_election.cc:288] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [CANDIDATE]: Term 1 election: Requested vote from peers c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591), 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:22.192147 20025 leader_election.cc:288] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 1 election: Requested vote from peers 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811), 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:22.192195 19900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e6457de95c64fb19ae1d294c436c8f7" candidate_uuid: "7d4b038952284c81a07423e42a349192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
I20240501 14:10:22.192289 19900 raft_consensus.cc:3037] T 2e6457de95c64fb19ae1d294c436c8f7 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.192301 19975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e6457de95c64fb19ae1d294c436c8f7" candidate_uuid: "7d4b038952284c81a07423e42a349192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
I20240501 14:10:22.192373 19975 raft_consensus.cc:3037] T 2e6457de95c64fb19ae1d294c436c8f7 P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.192456 19824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0888af56f2427c9387c3682f211a58" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4b038952284c81a07423e42a349192"
I20240501 14:10:22.192533 19824 raft_consensus.cc:3037] T 9d0888af56f2427c9387c3682f211a58 P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.193100 19975 raft_consensus.cc:2454] T 2e6457de95c64fb19ae1d294c436c8f7 P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d4b038952284c81a07423e42a349192 in term 1.
I20240501 14:10:22.193090 19824 raft_consensus.cc:2454] T 9d0888af56f2427c9387c3682f211a58 P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 1.
I20240501 14:10:22.193100 19900 raft_consensus.cc:2454] T 2e6457de95c64fb19ae1d294c436c8f7 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d4b038952284c81a07423e42a349192 in term 1.
I20240501 14:10:22.193423 19790 leader_election.cc:302] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4b038952284c81a07423e42a349192, 94100faea1434b27a8c6d129d1b562bc; no voters:
I20240501 14:10:22.193420 19975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0888af56f2427c9387c3682f211a58" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
I20240501 14:10:22.193572 19975 raft_consensus.cc:3037] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.193639 19866 leader_election.cc:302] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4b038952284c81a07423e42a349192, c4d2a3d7c38d4794a1ee78eaef2940d1; no voters:
I20240501 14:10:22.193794 20027 raft_consensus.cc:2781] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:22.193796 20025 raft_consensus.cc:2781] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:22.193857 20027 raft_consensus.cc:686] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [term 1 LEADER]: Becoming Leader. State: Replica: 7d4b038952284c81a07423e42a349192, State: Running, Role: LEADER
I20240501 14:10:22.194058 20027 consensus_queue.cc:237] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.194202 20025 raft_consensus.cc:686] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Becoming Leader. State: Replica: c4d2a3d7c38d4794a1ee78eaef2940d1, State: Running, Role: LEADER
I20240501 14:10:22.194355 20025 consensus_queue.cc:237] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.194366 19975 raft_consensus.cc:2454] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 1.
I20240501 14:10:22.196509 19719 catalog_manager.cc:5484] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d4b038952284c81a07423e42a349192 (127.16.13.193). New cstate: current_term: 1 leader_uuid: "7d4b038952284c81a07423e42a349192" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:22.196517 19720 catalog_manager.cc:5484] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194). New cstate: current_term: 1 leader_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:22.273816 20032 raft_consensus.cc:482] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:22.273977 20032 raft_consensus.cc:504] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.274376 20032 leader_election.cc:288] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811), c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591)
I20240501 14:10:22.274807 19900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603909ab31c7462c9e2c893184a5fc6c" candidate_uuid: "94100faea1434b27a8c6d129d1b562bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" is_pre_election: true
I20240501 14:10:22.274808 19824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603909ab31c7462c9e2c893184a5fc6c" candidate_uuid: "94100faea1434b27a8c6d129d1b562bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4b038952284c81a07423e42a349192" is_pre_election: true
I20240501 14:10:22.274935 19900 raft_consensus.cc:2454] T 603909ab31c7462c9e2c893184a5fc6c P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94100faea1434b27a8c6d129d1b562bc in term 0.
I20240501 14:10:22.274986 19824 raft_consensus.cc:2454] T 603909ab31c7462c9e2c893184a5fc6c P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94100faea1434b27a8c6d129d1b562bc in term 0.
I20240501 14:10:22.275238 19941 leader_election.cc:302] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4b038952284c81a07423e42a349192, 94100faea1434b27a8c6d129d1b562bc; no voters:
I20240501 14:10:22.275408 20032 raft_consensus.cc:2781] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:22.275460 20032 raft_consensus.cc:482] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:22.275487 20032 raft_consensus.cc:3037] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.276364 20032 raft_consensus.cc:504] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.276721 20032 leader_election.cc:288] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [CANDIDATE]: Term 1 election: Requested vote from peers 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811), c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591)
I20240501 14:10:22.277051 19900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603909ab31c7462c9e2c893184a5fc6c" candidate_uuid: "94100faea1434b27a8c6d129d1b562bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
I20240501 14:10:22.277048 19824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603909ab31c7462c9e2c893184a5fc6c" candidate_uuid: "94100faea1434b27a8c6d129d1b562bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4b038952284c81a07423e42a349192"
I20240501 14:10:22.277189 19900 raft_consensus.cc:3037] T 603909ab31c7462c9e2c893184a5fc6c P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.277189 19824 raft_consensus.cc:3037] T 603909ab31c7462c9e2c893184a5fc6c P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.278084 19900 raft_consensus.cc:2454] T 603909ab31c7462c9e2c893184a5fc6c P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94100faea1434b27a8c6d129d1b562bc in term 1.
I20240501 14:10:22.278084 19824 raft_consensus.cc:2454] T 603909ab31c7462c9e2c893184a5fc6c P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94100faea1434b27a8c6d129d1b562bc in term 1.
I20240501 14:10:22.278415 19941 leader_election.cc:302] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4b038952284c81a07423e42a349192, 94100faea1434b27a8c6d129d1b562bc; no voters:
I20240501 14:10:22.278560 20032 raft_consensus.cc:2781] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:22.278642 20032 raft_consensus.cc:686] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [term 1 LEADER]: Becoming Leader. State: Replica: 94100faea1434b27a8c6d129d1b562bc, State: Running, Role: LEADER
I20240501 14:10:22.278807 20032 consensus_queue.cc:237] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.280550 19719 catalog_manager.cc:5484] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 94100faea1434b27a8c6d129d1b562bc (127.16.13.195). New cstate: current_term: 1 leader_uuid: "94100faea1434b27a8c6d129d1b562bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: HEALTHY } } }
I20240501 14:10:22.389772 20025 raft_consensus.cc:482] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:22.389947 20025 raft_consensus.cc:504] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.390444 20025 leader_election.cc:288] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811), 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:22.390913 19975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a726c9cea76241df8a00d52d6175db7a" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94100faea1434b27a8c6d129d1b562bc" is_pre_election: true
I20240501 14:10:22.390913 19824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a726c9cea76241df8a00d52d6175db7a" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4b038952284c81a07423e42a349192" is_pre_election: true
I20240501 14:10:22.391093 19824 raft_consensus.cc:2454] T a726c9cea76241df8a00d52d6175db7a P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 0.
I20240501 14:10:22.391093 19975 raft_consensus.cc:2454] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 0.
I20240501 14:10:22.391407 19866 leader_election.cc:302] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4b038952284c81a07423e42a349192, c4d2a3d7c38d4794a1ee78eaef2940d1; no voters:
I20240501 14:10:22.391590 20025 raft_consensus.cc:2781] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:22.391646 20025 raft_consensus.cc:482] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:22.391692 20025 raft_consensus.cc:3037] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.392557 20025 raft_consensus.cc:504] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.392869 20025 leader_election.cc:288] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 1 election: Requested vote from peers 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811), 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:22.393347 19975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a726c9cea76241df8a00d52d6175db7a" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
I20240501 14:10:22.393347 19824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a726c9cea76241df8a00d52d6175db7a" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4b038952284c81a07423e42a349192"
I20240501 14:10:22.393502 19824 raft_consensus.cc:3037] T a726c9cea76241df8a00d52d6175db7a P 7d4b038952284c81a07423e42a349192 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.393502 19975 raft_consensus.cc:3037] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:22.394438 19824 raft_consensus.cc:2454] T a726c9cea76241df8a00d52d6175db7a P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 1.
I20240501 14:10:22.394438 19975 raft_consensus.cc:2454] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 1.
I20240501 14:10:22.394778 19866 leader_election.cc:302] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4b038952284c81a07423e42a349192, c4d2a3d7c38d4794a1ee78eaef2940d1; no voters:
I20240501 14:10:22.394927 20025 raft_consensus.cc:2781] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:22.395009 20025 raft_consensus.cc:686] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Becoming Leader. State: Replica: c4d2a3d7c38d4794a1ee78eaef2940d1, State: Running, Role: LEADER
I20240501 14:10:22.395222 20025 consensus_queue.cc:237] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.396991 19719 catalog_manager.cc:5484] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194). New cstate: current_term: 1 leader_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:22.442145 20025 consensus_queue.cc:1035] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:22.445887 20073 consensus_queue.cc:1035] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:22.446731 20027 consensus_queue.cc:1035] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:22.449985 20027 consensus_queue.cc:1035] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:22.473275 20073 consensus_queue.cc:1035] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:22.476176 20093 consensus_queue.cc:1035] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:22.498965 20073 consensus_queue.cc:1035] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:22.505856 20073 consensus_queue.cc:1035] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:22.509299 20032 consensus_queue.cc:1035] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:22.512254 20035 consensus_queue.cc:1035] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:22.529775 19822 raft_consensus.cc:1261] T 4b28da769e1f46e1b05bd25e8428316f P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:22.530498 19899 raft_consensus.cc:1261] T 2e6457de95c64fb19ae1d294c436c8f7 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:22.530530 19973 raft_consensus.cc:1261] T 2e6457de95c64fb19ae1d294c436c8f7 P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:22.530795 19898 raft_consensus.cc:1261] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:22.530851 19971 raft_consensus.cc:1261] T 352498cc20094372a957c030cebf375e P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:22.531077 19897 raft_consensus.cc:1261] T 352498cc20094372a957c030cebf375e P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:22.531514 20123 consensus_queue.cc:1035] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:22.531809 19968 raft_consensus.cc:1261] T 4b28da769e1f46e1b05bd25e8428316f P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:22.531816 20123 consensus_queue.cc:1035] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:22.532121 20123 consensus_queue.cc:1035] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:22.532446 19969 raft_consensus.cc:1261] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:22.530530 19972 raft_consensus.cc:1261] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:22.533537 20073 consensus_queue.cc:1035] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:22.534896 20025 consensus_queue.cc:1035] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:22.535130 20025 consensus_queue.cc:1035] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:22.535605 19821 raft_consensus.cc:1261] T 9d0888af56f2427c9387c3682f211a58 P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:22.535770 20067 consensus_queue.cc:1035] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:22.536051 20067 consensus_queue.cc:1035] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:22.536295 20067 consensus_queue.cc:1035] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:22.544028 20093 consensus_queue.cc:1035] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:22.555608 19900 raft_consensus.cc:1261] T 603909ab31c7462c9e2c893184a5fc6c P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:22.556608 20035 consensus_queue.cc:1035] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:22.564055 19967 raft_consensus.cc:1261] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:22.565152 19823 raft_consensus.cc:1261] T 603909ab31c7462c9e2c893184a5fc6c P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:22.566130 20093 consensus_queue.cc:1035] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:22.567095 19823 raft_consensus.cc:1261] T a726c9cea76241df8a00d52d6175db7a P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:22.567868 20120 consensus_queue.cc:1035] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:22.569404 20032 consensus_queue.cc:1035] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:22.587116 20110 mvcc.cc:205] Tried to move back new op lower bound from 7022889461868109824 to 7022889459870265344. Current Snapshot: MvccSnapshot[applied={T|T < 7022889461861683200}]
I20240501 14:10:22.602162 20087 mvcc.cc:205] Tried to move back new op lower bound from 7022889461848899584 to 7022889460509749248. Current Snapshot: MvccSnapshot[applied={T|T < 7022889461848899584}]
I20240501 14:10:22.744228 19775 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:10:22.794008 19896 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9d0888af56f2427c9387c3682f211a58"
dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
mode: GRACEFUL
new_leader_uuid: "94100faea1434b27a8c6d129d1b562bc"
from {username='slave'} at 127.0.0.1:56822
I20240501 14:10:22.794338 19896 raft_consensus.cc:595] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Received request to transfer leadership to TS 94100faea1434b27a8c6d129d1b562bc
I20240501 14:10:22.795028 19896 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a726c9cea76241df8a00d52d6175db7a"
dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
mode: GRACEFUL
new_leader_uuid: "94100faea1434b27a8c6d129d1b562bc"
from {username='slave'} at 127.0.0.1:56822
I20240501 14:10:22.795162 19896 raft_consensus.cc:595] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Received request to transfer leadership to TS 94100faea1434b27a8c6d129d1b562bc
I20240501 14:10:22.795508 19775 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240501 14:10:22.795634 19775 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240501 14:10:22.798897 19871 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Service unavailable: leader transfer in progress
W20240501 14:10:22.799526 19880 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Service unavailable: leader transfer in progress
W20240501 14:10:22.811084 19880 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Service unavailable: leader transfer in progress
W20240501 14:10:22.811674 19880 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Service unavailable: leader transfer in progress
W20240501 14:10:22.817288 19880 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Service unavailable: leader transfer in progress
W20240501 14:10:22.818415 19880 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Service unavailable: leader transfer in progress
W20240501 14:10:22.837906 19880 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Service unavailable: leader transfer in progress
W20240501 14:10:22.839195 19880 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Service unavailable: leader transfer in progress
W20240501 14:10:22.844341 19880 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Service unavailable: leader transfer in progress
W20240501 14:10:22.845621 19880 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Service unavailable: leader transfer in progress
W20240501 14:10:22.849968 19880 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Service unavailable: leader transfer in progress
W20240501 14:10:22.850443 19877 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Service unavailable: leader transfer in progress
I20240501 14:10:22.854449 20173 raft_consensus.cc:979] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1: : Instructing follower 94100faea1434b27a8c6d129d1b562bc to start an election
I20240501 14:10:22.854589 20174 raft_consensus.cc:1067] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Signalling peer 94100faea1434b27a8c6d129d1b562bc to start an election
W20240501 14:10:22.856444 19877 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Service unavailable: leader transfer in progress
I20240501 14:10:22.856433 20173 raft_consensus.cc:979] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1: : Instructing follower 94100faea1434b27a8c6d129d1b562bc to start an election
I20240501 14:10:22.856813 20174 raft_consensus.cc:1067] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Signalling peer 94100faea1434b27a8c6d129d1b562bc to start an election
W20240501 14:10:22.856844 19880 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Service unavailable: leader transfer in progress
W20240501 14:10:22.857702 19880 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Service unavailable: leader transfer in progress
W20240501 14:10:22.859716 19877 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Service unavailable: leader transfer in progress
W20240501 14:10:22.860217 19877 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Service unavailable: leader transfer in progress
W20240501 14:10:22.862897 19877 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Service unavailable: leader transfer in progress
W20240501 14:10:22.864243 19877 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Service unavailable: leader transfer in progress
W20240501 14:10:22.864773 19877 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Service unavailable: leader transfer in progress
I20240501 14:10:22.865916 19972 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9d0888af56f2427c9387c3682f211a58"
dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
from {username='slave'} at 127.0.0.1:52796
I20240501 14:10:22.866053 19972 raft_consensus.cc:482] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:22.866122 19972 raft_consensus.cc:3037] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:22.867331 19972 raft_consensus.cc:504] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.868017 19972 leader_election.cc:288] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [CANDIDATE]: Term 2 election: Requested vote from peers 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811), c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591)
I20240501 14:10:22.869026 19968 raft_consensus.cc:1226] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [term 2 FOLLOWER]: Rejecting Update request from peer c4d2a3d7c38d4794a1ee78eaef2940d1 for earlier term 1. Current term is 2. Ops: []
W20240501 14:10:22.875983 19877 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Service unavailable: leader transfer in progress
W20240501 14:10:22.876606 19877 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Service unavailable: leader transfer in progress
W20240501 14:10:22.877130 19877 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Service unavailable: leader transfer in progress
W20240501 14:10:22.877645 19877 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Service unavailable: leader transfer in progress
W20240501 14:10:22.878182 19877 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Service unavailable: leader transfer in progress
W20240501 14:10:22.878239 19880 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Service unavailable: leader transfer in progress
W20240501 14:10:22.878762 19880 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Service unavailable: leader transfer in progress
W20240501 14:10:22.879257 19880 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Service unavailable: leader transfer in progress
I20240501 14:10:22.880827 19896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0888af56f2427c9387c3682f211a58" candidate_uuid: "94100faea1434b27a8c6d129d1b562bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: true dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
I20240501 14:10:22.880959 19896 raft_consensus.cc:3032] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:10:22.881008 19896 raft_consensus.cc:729] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c4d2a3d7c38d4794a1ee78eaef2940d1, State: Running, Role: LEADER
I20240501 14:10:22.881178 19896 consensus_queue.cc:260] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 1.17, Last appended by leader: 17, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.881350 19896 raft_consensus.cc:3037] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:22.882560 19896 raft_consensus.cc:2454] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94100faea1434b27a8c6d129d1b562bc in term 2.
I20240501 14:10:22.883839 19972 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a726c9cea76241df8a00d52d6175db7a"
dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
from {username='slave'} at 127.0.0.1:52796
I20240501 14:10:22.883988 19972 raft_consensus.cc:482] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:22.884061 19972 raft_consensus.cc:3037] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:22.885113 19972 raft_consensus.cc:504] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.885711 19968 raft_consensus.cc:1226] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [term 2 FOLLOWER]: Rejecting Update request from peer c4d2a3d7c38d4794a1ee78eaef2940d1 for earlier term 1. Current term is 2. Ops: []
I20240501 14:10:22.885921 19823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0888af56f2427c9387c3682f211a58" candidate_uuid: "94100faea1434b27a8c6d129d1b562bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: true dest_uuid: "7d4b038952284c81a07423e42a349192"
I20240501 14:10:22.886130 19823 raft_consensus.cc:3037] T 9d0888af56f2427c9387c3682f211a58 P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:22.886298 19819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a726c9cea76241df8a00d52d6175db7a" candidate_uuid: "94100faea1434b27a8c6d129d1b562bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: true dest_uuid: "7d4b038952284c81a07423e42a349192"
I20240501 14:10:22.886412 19819 raft_consensus.cc:3037] T a726c9cea76241df8a00d52d6175db7a P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:22.886752 19939 leader_election.cc:302] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94100faea1434b27a8c6d129d1b562bc, c4d2a3d7c38d4794a1ee78eaef2940d1; no voters:
I20240501 14:10:22.886930 20179 raft_consensus.cc:2781] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:22.887019 20179 raft_consensus.cc:686] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [term 2 LEADER]: Becoming Leader. State: Replica: 94100faea1434b27a8c6d129d1b562bc, State: Running, Role: LEADER
I20240501 14:10:22.887151 20179 consensus_queue.cc:237] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.17, Last appended by leader: 17, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.887544 19819 raft_consensus.cc:2454] T a726c9cea76241df8a00d52d6175db7a P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94100faea1434b27a8c6d129d1b562bc in term 2.
I20240501 14:10:22.887665 19823 raft_consensus.cc:2454] T 9d0888af56f2427c9387c3682f211a58 P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94100faea1434b27a8c6d129d1b562bc in term 2.
I20240501 14:10:22.888932 19941 leader_election.cc:302] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4b038952284c81a07423e42a349192, 94100faea1434b27a8c6d129d1b562bc; no voters:
I20240501 14:10:22.889168 20168 raft_consensus.cc:2781] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:22.889307 20168 raft_consensus.cc:686] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [term 2 LEADER]: Becoming Leader. State: Replica: 94100faea1434b27a8c6d129d1b562bc, State: Running, Role: LEADER
I20240501 14:10:22.889462 19712 catalog_manager.cc:5484] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc reported cstate change: term changed from 1 to 2, leader changed from c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194) to 94100faea1434b27a8c6d129d1b562bc (127.16.13.195). New cstate: current_term: 2 leader_uuid: "94100faea1434b27a8c6d129d1b562bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: HEALTHY } } }
I20240501 14:10:22.889537 20168 consensus_queue.cc:237] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.15, Last appended by leader: 15, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
W20240501 14:10:22.892740 19877 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Service unavailable: leader transfer in progress
I20240501 14:10:22.892825 19895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a726c9cea76241df8a00d52d6175db7a" candidate_uuid: "94100faea1434b27a8c6d129d1b562bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: true dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
I20240501 14:10:22.892875 20178 consensus_queue.cc:1046] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 }, Status: INVALID_TERM, Last received: 1.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 14:10:22.893007 19895 raft_consensus.cc:3032] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:10:22.893051 19895 raft_consensus.cc:729] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c4d2a3d7c38d4794a1ee78eaef2940d1, State: Running, Role: LEADER
I20240501 14:10:22.893170 19895 consensus_queue.cc:260] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 15, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:22.893352 19895 raft_consensus.cc:3037] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:22.894368 19895 raft_consensus.cc:2454] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94100faea1434b27a8c6d129d1b562bc in term 2.
W20240501 14:10:22.894498 20174 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
W20240501 14:10:22.894760 19880 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Illegal state: replica c4d2a3d7c38d4794a1ee78eaef2940d1 is not leader of this config: current role FOLLOWER
I20240501 14:10:22.892825 19972 leader_election.cc:288] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [CANDIDATE]: Term 2 election: Requested vote from peers 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811), c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591)
I20240501 14:10:22.896188 19712 catalog_manager.cc:5484] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc reported cstate change: term changed from 1 to 2, leader changed from c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194) to 94100faea1434b27a8c6d129d1b562bc (127.16.13.195). New cstate: current_term: 2 leader_uuid: "94100faea1434b27a8c6d129d1b562bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: HEALTHY } } }
W20240501 14:10:22.897665 19880 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Illegal state: replica c4d2a3d7c38d4794a1ee78eaef2940d1 is not leader of this config: current role FOLLOWER
W20240501 14:10:22.898214 19880 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Illegal state: replica c4d2a3d7c38d4794a1ee78eaef2940d1 is not leader of this config: current role FOLLOWER
W20240501 14:10:22.898712 19880 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Illegal state: replica c4d2a3d7c38d4794a1ee78eaef2940d1 is not leader of this config: current role FOLLOWER
W20240501 14:10:22.900074 19880 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Illegal state: replica c4d2a3d7c38d4794a1ee78eaef2940d1 is not leader of this config: current role FOLLOWER
W20240501 14:10:22.900810 19804 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51484: Illegal state: replica 7d4b038952284c81a07423e42a349192 is not leader of this config: current role FOLLOWER
W20240501 14:10:22.900816 19871 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Illegal state: replica c4d2a3d7c38d4794a1ee78eaef2940d1 is not leader of this config: current role FOLLOWER
W20240501 14:10:22.900807 19877 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Illegal state: replica c4d2a3d7c38d4794a1ee78eaef2940d1 is not leader of this config: current role FOLLOWER
W20240501 14:10:22.900925 19880 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56808: Illegal state: replica c4d2a3d7c38d4794a1ee78eaef2940d1 is not leader of this config: current role FOLLOWER
W20240501 14:10:22.906920 19804 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51484: Illegal state: replica 7d4b038952284c81a07423e42a349192 is not leader of this config: current role FOLLOWER
W20240501 14:10:22.907794 19804 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51484: Illegal state: replica 7d4b038952284c81a07423e42a349192 is not leader of this config: current role FOLLOWER
W20240501 14:10:22.908934 19804 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51484: Illegal state: replica 7d4b038952284c81a07423e42a349192 is not leader of this config: current role FOLLOWER
W20240501 14:10:22.909103 19799 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51484: Illegal state: replica 7d4b038952284c81a07423e42a349192 is not leader of this config: current role FOLLOWER
W20240501 14:10:22.910143 19799 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51484: Illegal state: replica 7d4b038952284c81a07423e42a349192 is not leader of this config: current role FOLLOWER
I20240501 14:10:22.911057 19896 raft_consensus.cc:1261] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 2 index: 17. (index mismatch)
I20240501 14:10:22.911090 19820 raft_consensus.cc:1261] T a726c9cea76241df8a00d52d6175db7a P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 2 index: 17. (index mismatch)
I20240501 14:10:22.911409 20168 consensus_queue.cc:1035] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240501 14:10:22.911549 20179 consensus_queue.cc:1035] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20240501 14:10:22.913569 19799 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51484: Illegal state: replica 7d4b038952284c81a07423e42a349192 is not leader of this config: current role FOLLOWER
I20240501 14:10:22.916730 19896 raft_consensus.cc:1261] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 2 index: 19. (index mismatch)
I20240501 14:10:22.917901 19820 raft_consensus.cc:1261] T 9d0888af56f2427c9387c3682f211a58 P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 2 index: 19. (index mismatch)
I20240501 14:10:22.919157 20179 consensus_queue.cc:1035] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20240501 14:10:22.919483 20035 consensus_queue.cc:1035] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20240501 14:10:23.033522 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:10:23.037094 20182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:23.038475 20185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:23.038753 20183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:10:23.040095 16439 server_base.cc:1034] running on GCE node
I20240501 14:10:23.040329 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:10:23.040370 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:10:23.040387 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572623040387 us; error 0 us; skew 500 ppm
I20240501 14:10:23.041462 16439 webserver.cc:460] Webserver started at http://127.16.13.196:44627/ using document root <none> and password file <none>
I20240501 14:10:23.041596 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:10:23.041641 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:10:23.041698 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:10:23.042094 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-3-root/instance:
uuid: "fe34f459988040e291a748133a04015c"
format_stamp: "Formatted at 2024-05-01 14:10:23 on dist-test-slave-mgxc"
I20240501 14:10:23.043694 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 14:10:23.044657 20190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:23.044917 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:23.045014 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-3-root
uuid: "fe34f459988040e291a748133a04015c"
format_stamp: "Formatted at 2024-05-01 14:10:23 on dist-test-slave-mgxc"
I20240501 14:10:23.045109 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:10:23.052211 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:10:23.052608 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:10:23.052850 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:10:23.053283 16439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:10:23.053323 16439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:23.053373 16439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:10:23.053402 16439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:23.058367 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.196:35779
I20240501 14:10:23.058434 20253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.196:35779 every 8 connection(s)
I20240501 14:10:23.059229 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:10:23.061635 20258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:23.062456 20259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:23.063407 20261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:10:23.063629 20254 heartbeater.cc:347] Connected to a master server at 127.16.13.254:40095
I20240501 14:10:23.063731 20254 heartbeater.cc:464] Registering TS with master...
I20240501 14:10:23.064039 20254 heartbeater.cc:511] Master 127.16.13.254:40095 requested a full tablet report, sending...
I20240501 14:10:23.064211 16439 server_base.cc:1034] running on GCE node
I20240501 14:10:23.064404 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:10:23.064460 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:10:23.064483 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572623064483 us; error 0 us; skew 500 ppm
I20240501 14:10:23.064785 19718 ts_manager.cc:194] Registered new tserver with Master: fe34f459988040e291a748133a04015c (127.16.13.196:35779)
I20240501 14:10:23.065464 19718 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56454
I20240501 14:10:23.065799 16439 webserver.cc:460] Webserver started at http://127.16.13.197:36117/ using document root <none> and password file <none>
I20240501 14:10:23.065953 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:10:23.066004 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:10:23.066100 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:10:23.066519 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-4-root/instance:
uuid: "d0c0d1154175432ea3332f347685aa43"
format_stamp: "Formatted at 2024-05-01 14:10:23 on dist-test-slave-mgxc"
I20240501 14:10:23.068120 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 14:10:23.069109 20266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:23.069419 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:23.069494 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-4-root
uuid: "d0c0d1154175432ea3332f347685aa43"
format_stamp: "Formatted at 2024-05-01 14:10:23 on dist-test-slave-mgxc"
I20240501 14:10:23.069571 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-4-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-4-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-4-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:10:23.074366 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:10:23.074760 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:10:23.075071 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:10:23.075527 16439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:10:23.075572 16439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:23.075623 16439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:10:23.075651 16439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:23.081369 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.197:40045
I20240501 14:10:23.081462 20329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.197:40045 every 8 connection(s)
I20240501 14:10:23.082294 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:10:23.084941 20334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:23.085574 20333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:23.086422 20336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:10:23.086691 20330 heartbeater.cc:347] Connected to a master server at 127.16.13.254:40095
I20240501 14:10:23.086804 20330 heartbeater.cc:464] Registering TS with master...
I20240501 14:10:23.087071 20330 heartbeater.cc:511] Master 127.16.13.254:40095 requested a full tablet report, sending...
I20240501 14:10:23.087424 16439 server_base.cc:1034] running on GCE node
I20240501 14:10:23.087620 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:10:23.087668 16439 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:10:23.087690 16439 hybrid_clock.cc:648] HybridClock initialized: now 1714572623087690 us; error 0 us; skew 500 ppm
I20240501 14:10:23.087824 19718 ts_manager.cc:194] Registered new tserver with Master: d0c0d1154175432ea3332f347685aa43 (127.16.13.197:40045)
I20240501 14:10:23.088595 19718 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56460
I20240501 14:10:23.088923 16439 webserver.cc:460] Webserver started at http://127.16.13.198:37483/ using document root <none> and password file <none>
I20240501 14:10:23.089087 16439 fs_manager.cc:359] Metadata directory not provided
I20240501 14:10:23.089154 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:10:23.089237 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:10:23.089638 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-5-root/instance:
uuid: "5ce107fc401f4105be9947cf24c8c55e"
format_stamp: "Formatted at 2024-05-01 14:10:23 on dist-test-slave-mgxc"
I20240501 14:10:23.091182 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 14:10:23.092154 20341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:23.092386 16439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:23.092456 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-5-root
uuid: "5ce107fc401f4105be9947cf24c8c55e"
format_stamp: "Formatted at 2024-05-01 14:10:23 on dist-test-slave-mgxc"
I20240501 14:10:23.092530 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-5-root
metadata directory: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-5-root
1 data directories: /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-5-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:10:23.101087 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:10:23.101552 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:10:23.101876 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:10:23.102401 16439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:10:23.102454 16439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:23.102506 16439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:10:23.102535 16439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:23.107841 16439 rpc_server.cc:307] RPC server started. Bound to: 127.16.13.198:46107
I20240501 14:10:23.108000 20404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.13.198:46107 every 8 connection(s)
I20240501 14:10:23.112581 20405 heartbeater.cc:347] Connected to a master server at 127.16.13.254:40095
I20240501 14:10:23.112712 20405 heartbeater.cc:464] Registering TS with master...
I20240501 14:10:23.112931 20405 heartbeater.cc:511] Master 127.16.13.254:40095 requested a full tablet report, sending...
I20240501 14:10:23.113533 19718 ts_manager.cc:194] Registered new tserver with Master: 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198:46107)
I20240501 14:10:23.114081 19718 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56476
I20240501 14:10:23.753680 19974 consensus_queue.cc:237] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } }
I20240501 14:10:23.755307 19823 raft_consensus.cc:1261] T 603909ab31c7462c9e2c893184a5fc6c P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240501 14:10:23.755502 19895 raft_consensus.cc:1261] T 603909ab31c7462c9e2c893184a5fc6c P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240501 14:10:23.755755 20032 consensus_queue.cc:1035] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 14:10:23.756227 20032 consensus_queue.cc:1035] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 14:10:23.757500 20411 raft_consensus.cc:2932] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER fe34f459988040e291a748133a04015c (127.16.13.196) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } } }
I20240501 14:10:23.757879 19823 raft_consensus.cc:2932] T 603909ab31c7462c9e2c893184a5fc6c P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER fe34f459988040e291a748133a04015c (127.16.13.196) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } } }
I20240501 14:10:23.758069 19895 raft_consensus.cc:2932] T 603909ab31c7462c9e2c893184a5fc6c P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER fe34f459988040e291a748133a04015c (127.16.13.196) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } } }
W20240501 14:10:23.760593 19941 consensus_peers.cc:485] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc -> Peer fe34f459988040e291a748133a04015c (127.16.13.196:35779): Couldn't send request to peer fe34f459988040e291a748133a04015c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 603909ab31c7462c9e2c893184a5fc6c. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:23.760661 19718 catalog_manager.cc:5484] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc reported cstate change: config changed from index -1 to 26, NON_VOTER fe34f459988040e291a748133a04015c (127.16.13.196) added. New cstate: current_term: 1 leader_uuid: "94100faea1434b27a8c6d129d1b562bc" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:23.765477 19895 consensus_queue.cc:237] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } }
I20240501 14:10:23.767014 19823 raft_consensus.cc:1261] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 14:10:23.767290 19974 raft_consensus.cc:1261] T 515893894ce0478caa58378e6c2e047b P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 14:10:23.767582 20418 consensus_queue.cc:1035] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 14:10:23.767773 20419 consensus_queue.cc:1035] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 14:10:23.769265 20174 raft_consensus.cc:2932] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } } }
I20240501 14:10:23.769673 19823 raft_consensus.cc:2932] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } } }
W20240501 14:10:23.771431 19864 consensus_peers.cc:485] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 -> Peer 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198:46107): Couldn't send request to peer 5ce107fc401f4105be9947cf24c8c55e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 515893894ce0478caa58378e6c2e047b. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:23.771656 19974 raft_consensus.cc:2932] T 515893894ce0478caa58378e6c2e047b P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } } }
I20240501 14:10:23.771960 19968 consensus_queue.cc:237] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 2.25, Last appended by leader: 15, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } }
I20240501 14:10:23.772214 19720 catalog_manager.cc:5484] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 reported cstate change: config changed from index -1 to 27, NON_VOTER 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) added. New cstate: current_term: 1 leader_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:23.773424 19823 raft_consensus.cc:1261] T a726c9cea76241df8a00d52d6175db7a P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240501 14:10:23.773885 20422 consensus_queue.cc:1035] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 14:10:23.774019 19895 raft_consensus.cc:1261] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240501 14:10:23.774366 20422 consensus_queue.cc:1035] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 14:10:23.776314 20032 raft_consensus.cc:2932] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [term 2 LEADER]: Committing config change with OpId 2.26: config changed from index -1 to 26, NON_VOTER d0c0d1154175432ea3332f347685aa43 (127.16.13.197) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } } }
I20240501 14:10:23.776566 19895 raft_consensus.cc:2932] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Committing config change with OpId 2.26: config changed from index -1 to 26, NON_VOTER d0c0d1154175432ea3332f347685aa43 (127.16.13.197) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } } }
I20240501 14:10:23.776615 19823 raft_consensus.cc:2932] T a726c9cea76241df8a00d52d6175db7a P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Committing config change with OpId 2.26: config changed from index -1 to 26, NON_VOTER d0c0d1154175432ea3332f347685aa43 (127.16.13.197) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } } }
I20240501 14:10:23.778708 19895 consensus_queue.cc:237] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } }
I20240501 14:10:23.779162 19712 catalog_manager.cc:5484] T a726c9cea76241df8a00d52d6175db7a P 7d4b038952284c81a07423e42a349192 reported cstate change: config changed from index -1 to 26, NON_VOTER d0c0d1154175432ea3332f347685aa43 (127.16.13.197) added. New cstate: current_term: 2 leader_uuid: "94100faea1434b27a8c6d129d1b562bc" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } } }
I20240501 14:10:23.779857 19823 raft_consensus.cc:1261] T 4b28da769e1f46e1b05bd25e8428316f P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 14:10:23.780066 19974 raft_consensus.cc:1261] T 4b28da769e1f46e1b05bd25e8428316f P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 14:10:23.780565 20418 consensus_queue.cc:1035] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 14:10:23.780709 20419 consensus_queue.cc:1035] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 14:10:23.782265 20418 raft_consensus.cc:2932] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER d0c0d1154175432ea3332f347685aa43 (127.16.13.197) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } } }
W20240501 14:10:23.782907 19938 consensus_peers.cc:485] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc -> Peer d0c0d1154175432ea3332f347685aa43 (127.16.13.197:40045): Couldn't send request to peer d0c0d1154175432ea3332f347685aa43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a726c9cea76241df8a00d52d6175db7a. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:23.782841 19974 raft_consensus.cc:2932] T 4b28da769e1f46e1b05bd25e8428316f P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER d0c0d1154175432ea3332f347685aa43 (127.16.13.197) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } } }
I20240501 14:10:23.785216 19720 catalog_manager.cc:5484] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 reported cstate change: config changed from index -1 to 27, NON_VOTER d0c0d1154175432ea3332f347685aa43 (127.16.13.197) added. New cstate: current_term: 1 leader_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:10:23.785566 19863 consensus_peers.cc:485] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 -> Peer d0c0d1154175432ea3332f347685aa43 (127.16.13.197:40045): Couldn't send request to peer d0c0d1154175432ea3332f347685aa43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b28da769e1f46e1b05bd25e8428316f. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:23.785883 19823 raft_consensus.cc:2932] T 4b28da769e1f46e1b05bd25e8428316f P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER d0c0d1154175432ea3332f347685aa43 (127.16.13.197) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } } }
I20240501 14:10:23.788777 19823 consensus_queue.cc:237] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } }
I20240501 14:10:23.790313 19974 raft_consensus.cc:1261] T a47e39994ef3497e96bedd8a36133e00 P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 14:10:23.790366 19895 raft_consensus.cc:1261] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 14:10:23.790688 20426 consensus_queue.cc:1035] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 14:10:23.790881 20427 consensus_queue.cc:1035] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 14:10:23.792546 20426 raft_consensus.cc:2932] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER fe34f459988040e291a748133a04015c (127.16.13.196) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } } }
I20240501 14:10:23.792884 19895 raft_consensus.cc:2932] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER fe34f459988040e291a748133a04015c (127.16.13.196) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } } }
W20240501 14:10:23.793656 19790 consensus_peers.cc:485] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 -> Peer fe34f459988040e291a748133a04015c (127.16.13.196:35779): Couldn't send request to peer fe34f459988040e291a748133a04015c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a47e39994ef3497e96bedd8a36133e00. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:23.793856 19974 raft_consensus.cc:2932] T a47e39994ef3497e96bedd8a36133e00 P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER fe34f459988040e291a748133a04015c (127.16.13.196) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } } }
I20240501 14:10:23.795198 19712 catalog_manager.cc:5484] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1 reported cstate change: config changed from index -1 to 27, NON_VOTER fe34f459988040e291a748133a04015c (127.16.13.196) added. New cstate: current_term: 1 leader_uuid: "7d4b038952284c81a07423e42a349192" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } } }
I20240501 14:10:23.795625 19823 consensus_queue.cc:237] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } }
I20240501 14:10:23.796880 19895 raft_consensus.cc:1261] T 352498cc20094372a957c030cebf375e P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 14:10:23.796926 19974 raft_consensus.cc:1261] T 352498cc20094372a957c030cebf375e P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 14:10:23.797256 20124 consensus_queue.cc:1035] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 14:10:23.798327 20124 consensus_queue.cc:1035] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 14:10:23.798677 20124 raft_consensus.cc:2932] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } } }
I20240501 14:10:23.798964 19895 raft_consensus.cc:2932] T 352498cc20094372a957c030cebf375e P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } } }
I20240501 14:10:23.801198 19720 catalog_manager.cc:5484] T 352498cc20094372a957c030cebf375e P c4d2a3d7c38d4794a1ee78eaef2940d1 reported cstate change: config changed from index -1 to 27, NON_VOTER 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) added. New cstate: current_term: 1 leader_uuid: "7d4b038952284c81a07423e42a349192" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } } }
W20240501 14:10:23.802384 19788 consensus_peers.cc:485] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 -> Peer 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198:46107): Couldn't send request to peer 5ce107fc401f4105be9947cf24c8c55e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 352498cc20094372a957c030cebf375e. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:23.803885 19974 raft_consensus.cc:2932] T 352498cc20094372a957c030cebf375e P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } } }
I20240501 14:10:24.066373 20254 heartbeater.cc:503] Master 127.16.13.254:40095 was elected leader, sending a full tablet report...
I20240501 14:10:24.089465 20330 heartbeater.cc:503] Master 127.16.13.254:40095 was elected leader, sending a full tablet report...
I20240501 14:10:24.114981 20405 heartbeater.cc:503] Master 127.16.13.254:40095 was elected leader, sending a full tablet report...
I20240501 14:10:24.143342 20429 ts_tablet_manager.cc:902] T 603909ab31c7462c9e2c893184a5fc6c P fe34f459988040e291a748133a04015c: Initiating tablet copy from peer 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:24.143961 20429 tablet_copy_client.cc:308] T 603909ab31c7462c9e2c893184a5fc6c P fe34f459988040e291a748133a04015c: tablet copy: Beginning tablet copy session from remote peer at address 127.16.13.195:36685
I20240501 14:10:24.148005 19985 tablet_copy_service.cc:140] P 94100faea1434b27a8c6d129d1b562bc: Received BeginTabletCopySession request for tablet 603909ab31c7462c9e2c893184a5fc6c from peer fe34f459988040e291a748133a04015c ({username='slave'} at 127.0.0.1:52822)
I20240501 14:10:24.148149 19985 tablet_copy_service.cc:161] P 94100faea1434b27a8c6d129d1b562bc: Beginning new tablet copy session on tablet 603909ab31c7462c9e2c893184a5fc6c from peer fe34f459988040e291a748133a04015c at {username='slave'} at 127.0.0.1:52822: session id = fe34f459988040e291a748133a04015c-603909ab31c7462c9e2c893184a5fc6c
I20240501 14:10:24.149284 19985 tablet_copy_source_session.cc:205] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:24.150108 20429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 603909ab31c7462c9e2c893184a5fc6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:24.152215 20429 tablet_copy_client.cc:783] T 603909ab31c7462c9e2c893184a5fc6c P fe34f459988040e291a748133a04015c: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:24.152426 20429 tablet_copy_client.cc:647] T 603909ab31c7462c9e2c893184a5fc6c P fe34f459988040e291a748133a04015c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:24.153416 20433 tablet_copy_client.cc:1002] T 603909ab31c7462c9e2c893184a5fc6c P fe34f459988040e291a748133a04015c: tablet copy: Injecting latency into file download: 2000
I20240501 14:10:24.210927 20435 ts_tablet_manager.cc:902] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e: Initiating tablet copy from peer c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591)
I20240501 14:10:24.211704 20435 tablet_copy_client.cc:308] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e: tablet copy: Beginning tablet copy session from remote peer at address 127.16.13.194:39591
I20240501 14:10:24.216423 19910 tablet_copy_service.cc:140] P c4d2a3d7c38d4794a1ee78eaef2940d1: Received BeginTabletCopySession request for tablet 515893894ce0478caa58378e6c2e047b from peer 5ce107fc401f4105be9947cf24c8c55e ({username='slave'} at 127.0.0.1:56836)
I20240501 14:10:24.216569 19910 tablet_copy_service.cc:161] P c4d2a3d7c38d4794a1ee78eaef2940d1: Beginning new tablet copy session on tablet 515893894ce0478caa58378e6c2e047b from peer 5ce107fc401f4105be9947cf24c8c55e at {username='slave'} at 127.0.0.1:56836: session id = 5ce107fc401f4105be9947cf24c8c55e-515893894ce0478caa58378e6c2e047b
I20240501 14:10:24.217738 19910 tablet_copy_source_session.cc:205] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:24.223975 20435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 515893894ce0478caa58378e6c2e047b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:24.226279 20435 tablet_copy_client.cc:783] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:24.226452 20435 tablet_copy_client.cc:647] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:24.227749 20439 tablet_copy_client.cc:1002] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e: tablet copy: Injecting latency into file download: 2000
I20240501 14:10:24.242527 20441 ts_tablet_manager.cc:902] T 4b28da769e1f46e1b05bd25e8428316f P d0c0d1154175432ea3332f347685aa43: Initiating tablet copy from peer c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591)
I20240501 14:10:24.243368 20441 tablet_copy_client.cc:308] T 4b28da769e1f46e1b05bd25e8428316f P d0c0d1154175432ea3332f347685aa43: tablet copy: Beginning tablet copy session from remote peer at address 127.16.13.194:39591
I20240501 14:10:24.249759 19910 tablet_copy_service.cc:140] P c4d2a3d7c38d4794a1ee78eaef2940d1: Received BeginTabletCopySession request for tablet 4b28da769e1f46e1b05bd25e8428316f from peer d0c0d1154175432ea3332f347685aa43 ({username='slave'} at 127.0.0.1:56846)
I20240501 14:10:24.249872 19910 tablet_copy_service.cc:161] P c4d2a3d7c38d4794a1ee78eaef2940d1: Beginning new tablet copy session on tablet 4b28da769e1f46e1b05bd25e8428316f from peer d0c0d1154175432ea3332f347685aa43 at {username='slave'} at 127.0.0.1:56846: session id = d0c0d1154175432ea3332f347685aa43-4b28da769e1f46e1b05bd25e8428316f
I20240501 14:10:24.250892 19910 tablet_copy_source_session.cc:205] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:24.251801 20441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b28da769e1f46e1b05bd25e8428316f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:24.254225 20441 tablet_copy_client.cc:783] T 4b28da769e1f46e1b05bd25e8428316f P d0c0d1154175432ea3332f347685aa43: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:24.254465 20441 tablet_copy_client.cc:647] T 4b28da769e1f46e1b05bd25e8428316f P d0c0d1154175432ea3332f347685aa43: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:24.255980 20445 tablet_copy_client.cc:1002] T 4b28da769e1f46e1b05bd25e8428316f P d0c0d1154175432ea3332f347685aa43: tablet copy: Injecting latency into file download: 2000
I20240501 14:10:24.260636 20447 ts_tablet_manager.cc:902] T a726c9cea76241df8a00d52d6175db7a P d0c0d1154175432ea3332f347685aa43: Initiating tablet copy from peer 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:24.261394 20447 tablet_copy_client.cc:308] T a726c9cea76241df8a00d52d6175db7a P d0c0d1154175432ea3332f347685aa43: tablet copy: Beginning tablet copy session from remote peer at address 127.16.13.195:36685
I20240501 14:10:24.266324 19985 tablet_copy_service.cc:140] P 94100faea1434b27a8c6d129d1b562bc: Received BeginTabletCopySession request for tablet a726c9cea76241df8a00d52d6175db7a from peer d0c0d1154175432ea3332f347685aa43 ({username='slave'} at 127.0.0.1:52834)
I20240501 14:10:24.266466 19985 tablet_copy_service.cc:161] P 94100faea1434b27a8c6d129d1b562bc: Beginning new tablet copy session on tablet a726c9cea76241df8a00d52d6175db7a from peer d0c0d1154175432ea3332f347685aa43 at {username='slave'} at 127.0.0.1:52834: session id = d0c0d1154175432ea3332f347685aa43-a726c9cea76241df8a00d52d6175db7a
I20240501 14:10:24.267262 20452 ts_tablet_manager.cc:902] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e: Initiating tablet copy from peer 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811)
I20240501 14:10:24.267874 20452 tablet_copy_client.cc:308] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e: tablet copy: Beginning tablet copy session from remote peer at address 127.16.13.193:38811
I20240501 14:10:24.267949 19985 tablet_copy_source_session.cc:205] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:24.269064 20447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a726c9cea76241df8a00d52d6175db7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:24.271665 20447 tablet_copy_client.cc:783] T a726c9cea76241df8a00d52d6175db7a P d0c0d1154175432ea3332f347685aa43: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:24.271936 20447 tablet_copy_client.cc:647] T a726c9cea76241df8a00d52d6175db7a P d0c0d1154175432ea3332f347685aa43: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:24.272974 19834 tablet_copy_service.cc:140] P 7d4b038952284c81a07423e42a349192: Received BeginTabletCopySession request for tablet 352498cc20094372a957c030cebf375e from peer 5ce107fc401f4105be9947cf24c8c55e ({username='slave'} at 127.0.0.1:51512)
I20240501 14:10:24.273077 19834 tablet_copy_service.cc:161] P 7d4b038952284c81a07423e42a349192: Beginning new tablet copy session on tablet 352498cc20094372a957c030cebf375e from peer 5ce107fc401f4105be9947cf24c8c55e at {username='slave'} at 127.0.0.1:51512: session id = 5ce107fc401f4105be9947cf24c8c55e-352498cc20094372a957c030cebf375e
I20240501 14:10:24.273670 20450 tablet_copy_client.cc:1002] T a726c9cea76241df8a00d52d6175db7a P d0c0d1154175432ea3332f347685aa43: tablet copy: Injecting latency into file download: 2000
I20240501 14:10:24.273999 19834 tablet_copy_source_session.cc:205] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:24.274866 20452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 352498cc20094372a957c030cebf375e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:24.277690 20452 tablet_copy_client.cc:783] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:24.277912 20452 tablet_copy_client.cc:647] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:24.280858 20453 tablet_copy_client.cc:1002] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e: tablet copy: Injecting latency into file download: 2000
I20240501 14:10:24.389084 20457 ts_tablet_manager.cc:902] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c: Initiating tablet copy from peer 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811)
I20240501 14:10:24.389691 20457 tablet_copy_client.cc:308] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c: tablet copy: Beginning tablet copy session from remote peer at address 127.16.13.193:38811
I20240501 14:10:24.394348 19834 tablet_copy_service.cc:140] P 7d4b038952284c81a07423e42a349192: Received BeginTabletCopySession request for tablet a47e39994ef3497e96bedd8a36133e00 from peer fe34f459988040e291a748133a04015c ({username='slave'} at 127.0.0.1:51514)
I20240501 14:10:24.394474 19834 tablet_copy_service.cc:161] P 7d4b038952284c81a07423e42a349192: Beginning new tablet copy session on tablet a47e39994ef3497e96bedd8a36133e00 from peer fe34f459988040e291a748133a04015c at {username='slave'} at 127.0.0.1:51514: session id = fe34f459988040e291a748133a04015c-a47e39994ef3497e96bedd8a36133e00
I20240501 14:10:24.395449 19834 tablet_copy_source_session.cc:205] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:24.396445 20457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a47e39994ef3497e96bedd8a36133e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:24.399560 20457 tablet_copy_client.cc:783] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:24.399760 20457 tablet_copy_client.cc:647] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:24.400759 20461 tablet_copy_client.cc:1002] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c: tablet copy: Injecting latency into file download: 2000
I20240501 14:10:24.795892 19775 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:10:24.802902 19823 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f796e2ca050e4666b9eb2dfd8a9ee2ff"
dest_uuid: "7d4b038952284c81a07423e42a349192"
mode: GRACEFUL
new_leader_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
from {username='slave'} at 127.0.0.1:51516
I20240501 14:10:24.803056 19823 raft_consensus.cc:595] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 1 LEADER]: Received request to transfer leadership to TS c4d2a3d7c38d4794a1ee78eaef2940d1
I20240501 14:10:24.803382 19775 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 14:10:24.803539 19775 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:10:24.897670 20464 raft_consensus.cc:979] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192: : Instructing follower c4d2a3d7c38d4794a1ee78eaef2940d1 to start an election
I20240501 14:10:24.897809 20464 raft_consensus.cc:1067] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 1 LEADER]: Signalling peer c4d2a3d7c38d4794a1ee78eaef2940d1 to start an election
I20240501 14:10:24.898221 19896 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f796e2ca050e4666b9eb2dfd8a9ee2ff"
dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
from {username='slave'} at 127.0.0.1:56780
I20240501 14:10:24.898370 19896 raft_consensus.cc:482] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:24.898453 19896 raft_consensus.cc:3037] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:24.899603 19896 raft_consensus.cc:504] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:24.899971 19896 leader_election.cc:288] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 2 election: Requested vote from peers 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811), 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:24.900193 19823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f796e2ca050e4666b9eb2dfd8a9ee2ff" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: true dest_uuid: "7d4b038952284c81a07423e42a349192"
I20240501 14:10:24.900303 19823 raft_consensus.cc:3032] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:10:24.900270 19974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f796e2ca050e4666b9eb2dfd8a9ee2ff" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: true dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
I20240501 14:10:24.900374 19823 raft_consensus.cc:729] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7d4b038952284c81a07423e42a349192, State: Running, Role: LEADER
I20240501 14:10:24.900413 19974 raft_consensus.cc:3037] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:24.900496 19823 consensus_queue.cc:260] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 24, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:24.900664 19823 raft_consensus.cc:3037] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:24.901387 19974 raft_consensus.cc:2454] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 2.
I20240501 14:10:24.901401 19823 raft_consensus.cc:2454] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 2.
I20240501 14:10:24.901682 19866 leader_election.cc:302] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94100faea1434b27a8c6d129d1b562bc, c4d2a3d7c38d4794a1ee78eaef2940d1; no voters:
I20240501 14:10:24.901859 20466 raft_consensus.cc:2781] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:24.902076 20466 raft_consensus.cc:686] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 LEADER]: Becoming Leader. State: Replica: c4d2a3d7c38d4794a1ee78eaef2940d1, State: Running, Role: LEADER
I20240501 14:10:24.902220 20466 consensus_queue.cc:237] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 24, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:24.904253 19718 catalog_manager.cc:5484] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 reported cstate change: term changed from 1 to 2, leader changed from 7d4b038952284c81a07423e42a349192 (127.16.13.193) to c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194). New cstate: current_term: 2 leader_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:25.349081 19974 raft_consensus.cc:1261] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 2 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20240501 14:10:25.349480 20466 consensus_queue.cc:1035] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240501 14:10:25.352864 19819 raft_consensus.cc:1261] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20240501 14:10:25.353253 20419 consensus_queue.cc:1035] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240501 14:10:26.154002 20429 tablet_copy_client.cc:523] T 603909ab31c7462c9e2c893184a5fc6c P fe34f459988040e291a748133a04015c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:26.155496 20429 tablet_bootstrap.cc:492] T 603909ab31c7462c9e2c893184a5fc6c P fe34f459988040e291a748133a04015c: Bootstrap starting.
I20240501 14:10:26.176810 20429 tablet_bootstrap.cc:492] T 603909ab31c7462c9e2c893184a5fc6c P fe34f459988040e291a748133a04015c: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=83 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:26.177162 20429 tablet_bootstrap.cc:492] T 603909ab31c7462c9e2c893184a5fc6c P fe34f459988040e291a748133a04015c: Bootstrap complete.
I20240501 14:10:26.177670 20429 ts_tablet_manager.cc:1367] T 603909ab31c7462c9e2c893184a5fc6c P fe34f459988040e291a748133a04015c: Time spent bootstrapping tablet: real 0.022s user 0.019s sys 0.004s
I20240501 14:10:26.178198 20429 raft_consensus.cc:348] T 603909ab31c7462c9e2c893184a5fc6c P fe34f459988040e291a748133a04015c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } }
I20240501 14:10:26.178339 20429 raft_consensus.cc:729] T 603909ab31c7462c9e2c893184a5fc6c P fe34f459988040e291a748133a04015c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fe34f459988040e291a748133a04015c, State: Initialized, Role: LEARNER
I20240501 14:10:26.178434 20429 consensus_queue.cc:260] T 603909ab31c7462c9e2c893184a5fc6c P fe34f459988040e291a748133a04015c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } }
I20240501 14:10:26.178804 20429 ts_tablet_manager.cc:1398] T 603909ab31c7462c9e2c893184a5fc6c P fe34f459988040e291a748133a04015c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:26.179324 19985 tablet_copy_service.cc:342] P 94100faea1434b27a8c6d129d1b562bc: Request end of tablet copy session fe34f459988040e291a748133a04015c-603909ab31c7462c9e2c893184a5fc6c received from {username='slave'} at 127.0.0.1:52822
I20240501 14:10:26.179421 19985 tablet_copy_service.cc:433] P 94100faea1434b27a8c6d129d1b562bc: ending tablet copy session fe34f459988040e291a748133a04015c-603909ab31c7462c9e2c893184a5fc6c on tablet 603909ab31c7462c9e2c893184a5fc6c with peer fe34f459988040e291a748133a04015c
I20240501 14:10:26.228782 20435 tablet_copy_client.cc:523] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:26.230612 20435 tablet_bootstrap.cc:492] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e: Bootstrap starting.
I20240501 14:10:26.252832 20435 tablet_bootstrap.cc:492] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:26.253152 20435 tablet_bootstrap.cc:492] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e: Bootstrap complete.
I20240501 14:10:26.253367 20435 ts_tablet_manager.cc:1367] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e: Time spent bootstrapping tablet: real 0.023s user 0.018s sys 0.004s
I20240501 14:10:26.253804 20435 raft_consensus.cc:348] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } }
I20240501 14:10:26.253911 20435 raft_consensus.cc:729] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5ce107fc401f4105be9947cf24c8c55e, State: Initialized, Role: LEARNER
I20240501 14:10:26.253988 20435 consensus_queue.cc:260] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } }
I20240501 14:10:26.254319 20435 ts_tablet_manager.cc:1398] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:10:26.254727 19910 tablet_copy_service.cc:342] P c4d2a3d7c38d4794a1ee78eaef2940d1: Request end of tablet copy session 5ce107fc401f4105be9947cf24c8c55e-515893894ce0478caa58378e6c2e047b received from {username='slave'} at 127.0.0.1:56836
I20240501 14:10:26.254812 19910 tablet_copy_service.cc:433] P c4d2a3d7c38d4794a1ee78eaef2940d1: ending tablet copy session 5ce107fc401f4105be9947cf24c8c55e-515893894ce0478caa58378e6c2e047b on tablet 515893894ce0478caa58378e6c2e047b with peer 5ce107fc401f4105be9947cf24c8c55e
I20240501 14:10:26.256297 20441 tablet_copy_client.cc:523] T 4b28da769e1f46e1b05bd25e8428316f P d0c0d1154175432ea3332f347685aa43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:26.257536 20441 tablet_bootstrap.cc:492] T 4b28da769e1f46e1b05bd25e8428316f P d0c0d1154175432ea3332f347685aa43: Bootstrap starting.
I20240501 14:10:26.273923 20447 tablet_copy_client.cc:523] T a726c9cea76241df8a00d52d6175db7a P d0c0d1154175432ea3332f347685aa43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:26.275518 20447 tablet_bootstrap.cc:492] T a726c9cea76241df8a00d52d6175db7a P d0c0d1154175432ea3332f347685aa43: Bootstrap starting.
I20240501 14:10:26.281519 20441 tablet_bootstrap.cc:492] T 4b28da769e1f46e1b05bd25e8428316f P d0c0d1154175432ea3332f347685aa43: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=105 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:26.281862 20441 tablet_bootstrap.cc:492] T 4b28da769e1f46e1b05bd25e8428316f P d0c0d1154175432ea3332f347685aa43: Bootstrap complete.
I20240501 14:10:26.282076 20441 ts_tablet_manager.cc:1367] T 4b28da769e1f46e1b05bd25e8428316f P d0c0d1154175432ea3332f347685aa43: Time spent bootstrapping tablet: real 0.025s user 0.023s sys 0.000s
I20240501 14:10:26.282611 20441 raft_consensus.cc:348] T 4b28da769e1f46e1b05bd25e8428316f P d0c0d1154175432ea3332f347685aa43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } }
I20240501 14:10:26.282824 20441 raft_consensus.cc:729] T 4b28da769e1f46e1b05bd25e8428316f P d0c0d1154175432ea3332f347685aa43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d0c0d1154175432ea3332f347685aa43, State: Initialized, Role: LEARNER
I20240501 14:10:26.282986 20441 consensus_queue.cc:260] T 4b28da769e1f46e1b05bd25e8428316f P d0c0d1154175432ea3332f347685aa43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } }
I20240501 14:10:26.283128 20452 tablet_copy_client.cc:523] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:26.283366 20441 ts_tablet_manager.cc:1398] T 4b28da769e1f46e1b05bd25e8428316f P d0c0d1154175432ea3332f347685aa43: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 14:10:26.284276 19910 tablet_copy_service.cc:342] P c4d2a3d7c38d4794a1ee78eaef2940d1: Request end of tablet copy session d0c0d1154175432ea3332f347685aa43-4b28da769e1f46e1b05bd25e8428316f received from {username='slave'} at 127.0.0.1:56846
I20240501 14:10:26.284379 19910 tablet_copy_service.cc:433] P c4d2a3d7c38d4794a1ee78eaef2940d1: ending tablet copy session d0c0d1154175432ea3332f347685aa43-4b28da769e1f46e1b05bd25e8428316f on tablet 4b28da769e1f46e1b05bd25e8428316f with peer d0c0d1154175432ea3332f347685aa43
I20240501 14:10:26.284632 20452 tablet_bootstrap.cc:492] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e: Bootstrap starting.
I20240501 14:10:26.288275 20304 raft_consensus.cc:1203] T 4b28da769e1f46e1b05bd25e8428316f P d0c0d1154175432ea3332f347685aa43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240501 14:10:26.303210 20447 tablet_bootstrap.cc:492] T a726c9cea76241df8a00d52d6175db7a P d0c0d1154175432ea3332f347685aa43: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=96 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:26.303620 20447 tablet_bootstrap.cc:492] T a726c9cea76241df8a00d52d6175db7a P d0c0d1154175432ea3332f347685aa43: Bootstrap complete.
I20240501 14:10:26.303887 20447 ts_tablet_manager.cc:1367] T a726c9cea76241df8a00d52d6175db7a P d0c0d1154175432ea3332f347685aa43: Time spent bootstrapping tablet: real 0.028s user 0.014s sys 0.006s
I20240501 14:10:26.304358 20447 raft_consensus.cc:348] T a726c9cea76241df8a00d52d6175db7a P d0c0d1154175432ea3332f347685aa43 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } }
I20240501 14:10:26.304579 20447 raft_consensus.cc:729] T a726c9cea76241df8a00d52d6175db7a P d0c0d1154175432ea3332f347685aa43 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d0c0d1154175432ea3332f347685aa43, State: Initialized, Role: LEARNER
I20240501 14:10:26.304708 20447 consensus_queue.cc:260] T a726c9cea76241df8a00d52d6175db7a P d0c0d1154175432ea3332f347685aa43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } }
I20240501 14:10:26.304975 20447 ts_tablet_manager.cc:1398] T a726c9cea76241df8a00d52d6175db7a P d0c0d1154175432ea3332f347685aa43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:26.305684 19985 tablet_copy_service.cc:342] P 94100faea1434b27a8c6d129d1b562bc: Request end of tablet copy session d0c0d1154175432ea3332f347685aa43-a726c9cea76241df8a00d52d6175db7a received from {username='slave'} at 127.0.0.1:52834
I20240501 14:10:26.305763 19985 tablet_copy_service.cc:433] P 94100faea1434b27a8c6d129d1b562bc: ending tablet copy session d0c0d1154175432ea3332f347685aa43-a726c9cea76241df8a00d52d6175db7a on tablet a726c9cea76241df8a00d52d6175db7a with peer d0c0d1154175432ea3332f347685aa43
I20240501 14:10:26.308362 20452 tablet_bootstrap.cc:492] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=103 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:26.308663 20452 tablet_bootstrap.cc:492] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e: Bootstrap complete.
I20240501 14:10:26.308854 20452 ts_tablet_manager.cc:1367] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e: Time spent bootstrapping tablet: real 0.024s user 0.019s sys 0.003s
I20240501 14:10:26.309222 20452 raft_consensus.cc:348] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } }
I20240501 14:10:26.309326 20452 raft_consensus.cc:729] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5ce107fc401f4105be9947cf24c8c55e, State: Initialized, Role: LEARNER
I20240501 14:10:26.309423 20452 consensus_queue.cc:260] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } }
I20240501 14:10:26.309646 20452 ts_tablet_manager.cc:1398] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:26.310112 19834 tablet_copy_service.cc:342] P 7d4b038952284c81a07423e42a349192: Request end of tablet copy session 5ce107fc401f4105be9947cf24c8c55e-352498cc20094372a957c030cebf375e received from {username='slave'} at 127.0.0.1:51512
I20240501 14:10:26.310194 19834 tablet_copy_service.cc:433] P 7d4b038952284c81a07423e42a349192: ending tablet copy session 5ce107fc401f4105be9947cf24c8c55e-352498cc20094372a957c030cebf375e on tablet 352498cc20094372a957c030cebf375e with peer 5ce107fc401f4105be9947cf24c8c55e
I20240501 14:10:26.396780 20379 raft_consensus.cc:1203] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240501 14:10:26.401031 20457 tablet_copy_client.cc:523] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:26.402524 20457 tablet_bootstrap.cc:492] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c: Bootstrap starting.
I20240501 14:10:26.425037 20457 tablet_bootstrap.cc:492] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=105 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:26.425370 20457 tablet_bootstrap.cc:492] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c: Bootstrap complete.
I20240501 14:10:26.425575 20457 ts_tablet_manager.cc:1367] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c: Time spent bootstrapping tablet: real 0.023s user 0.016s sys 0.005s
I20240501 14:10:26.426092 20457 raft_consensus.cc:348] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } }
I20240501 14:10:26.426214 20457 raft_consensus.cc:729] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fe34f459988040e291a748133a04015c, State: Initialized, Role: LEARNER
I20240501 14:10:26.426330 20457 consensus_queue.cc:260] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } }
I20240501 14:10:26.426631 20457 ts_tablet_manager.cc:1398] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 14:10:26.427093 19834 tablet_copy_service.cc:342] P 7d4b038952284c81a07423e42a349192: Request end of tablet copy session fe34f459988040e291a748133a04015c-a47e39994ef3497e96bedd8a36133e00 received from {username='slave'} at 127.0.0.1:51514
I20240501 14:10:26.427196 19834 tablet_copy_service.cc:433] P 7d4b038952284c81a07423e42a349192: ending tablet copy session fe34f459988040e291a748133a04015c-a47e39994ef3497e96bedd8a36133e00 on tablet a47e39994ef3497e96bedd8a36133e00 with peer fe34f459988040e291a748133a04015c
I20240501 14:10:26.613905 20379 raft_consensus.cc:1203] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240501 14:10:26.634773 20228 raft_consensus.cc:1203] T 603909ab31c7462c9e2c893184a5fc6c P fe34f459988040e291a748133a04015c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20240501 14:10:26.759728 20304 raft_consensus.cc:1203] T a726c9cea76241df8a00d52d6175db7a P d0c0d1154175432ea3332f347685aa43 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.25->[2.26-2.26] Dedup: 2.26->[]
I20240501 14:10:26.803789 19775 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:10:26.805131 19775 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:10:26.805266 19775 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:10:26.867846 20466 raft_consensus.cc:1050] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1: attempting to promote NON_VOTER d0c0d1154175432ea3332f347685aa43 to VOTER
I20240501 14:10:26.868263 20466 consensus_queue.cc:237] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } }
I20240501 14:10:26.869424 20304 raft_consensus.cc:1261] T 4b28da769e1f46e1b05bd25e8428316f P d0c0d1154175432ea3332f347685aa43 [term 1 LEARNER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 14:10:26.869575 19974 raft_consensus.cc:1261] T 4b28da769e1f46e1b05bd25e8428316f P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 14:10:26.869637 19819 raft_consensus.cc:1261] T 4b28da769e1f46e1b05bd25e8428316f P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 14:10:26.869822 20489 consensus_queue.cc:1035] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:10:26.870101 20466 consensus_queue.cc:1035] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:10:26.870258 20492 consensus_queue.cc:1035] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:10:26.871842 20466 raft_consensus.cc:2932] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, d0c0d1154175432ea3332f347685aa43 (127.16.13.197) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:26.872613 19823 raft_consensus.cc:2932] T 4b28da769e1f46e1b05bd25e8428316f P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, d0c0d1154175432ea3332f347685aa43 (127.16.13.197) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:26.873528 20304 raft_consensus.cc:2932] T 4b28da769e1f46e1b05bd25e8428316f P d0c0d1154175432ea3332f347685aa43 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, d0c0d1154175432ea3332f347685aa43 (127.16.13.197) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:26.874974 19718 catalog_manager.cc:5484] T 4b28da769e1f46e1b05bd25e8428316f P 7d4b038952284c81a07423e42a349192 reported cstate change: config changed from index 27 to 28, d0c0d1154175432ea3332f347685aa43 (127.16.13.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:26.875645 19974 raft_consensus.cc:2932] T 4b28da769e1f46e1b05bd25e8428316f P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, d0c0d1154175432ea3332f347685aa43 (127.16.13.197) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:26.879571 19895 consensus_queue.cc:237] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } }
I20240501 14:10:26.880749 19968 raft_consensus.cc:1261] T 4b28da769e1f46e1b05bd25e8428316f P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 14:10:26.880766 20304 raft_consensus.cc:1261] T 4b28da769e1f46e1b05bd25e8428316f P d0c0d1154175432ea3332f347685aa43 [term 1 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 14:10:26.881094 20489 consensus_queue.cc:1035] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:10:26.881299 20492 consensus_queue.cc:1035] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:10:26.882714 20489 raft_consensus.cc:2932] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 7d4b038952284c81a07423e42a349192 (127.16.13.193) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:26.883054 20304 raft_consensus.cc:2932] T 4b28da769e1f46e1b05bd25e8428316f P d0c0d1154175432ea3332f347685aa43 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 7d4b038952284c81a07423e42a349192 (127.16.13.193) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:26.883229 19968 raft_consensus.cc:2932] T 4b28da769e1f46e1b05bd25e8428316f P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 7d4b038952284c81a07423e42a349192 (127.16.13.193) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:26.884856 19706 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4b28da769e1f46e1b05bd25e8428316f with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:10:26.885443 19714 catalog_manager.cc:5484] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 reported cstate change: config changed from index 28 to 29, VOTER 7d4b038952284c81a07423e42a349192 (127.16.13.193) evicted. New cstate: current_term: 1 leader_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:10:26.888283 19806 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b28da769e1f46e1b05bd25e8428316f with delete_type TABLET_DATA_TOMBSTONED (TS 7d4b038952284c81a07423e42a349192 not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:51456
I20240501 14:10:26.888660 20501 tablet_replica.cc:330] T 4b28da769e1f46e1b05bd25e8428316f P 7d4b038952284c81a07423e42a349192: stopping tablet replica
I20240501 14:10:26.888798 20501 raft_consensus.cc:2229] T 4b28da769e1f46e1b05bd25e8428316f P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:26.888976 20501 raft_consensus.cc:2258] T 4b28da769e1f46e1b05bd25e8428316f P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:26.889742 20501 ts_tablet_manager.cc:1875] T 4b28da769e1f46e1b05bd25e8428316f P 7d4b038952284c81a07423e42a349192: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:26.892228 20501 ts_tablet_manager.cc:1888] T 4b28da769e1f46e1b05bd25e8428316f P 7d4b038952284c81a07423e42a349192: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20240501 14:10:26.892355 20501 log.cc:1196] T 4b28da769e1f46e1b05bd25e8428316f P 7d4b038952284c81a07423e42a349192: Deleting WAL directory at /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-0-root/wals/4b28da769e1f46e1b05bd25e8428316f
I20240501 14:10:26.892832 19708 catalog_manager.cc:4830] TS 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811): tablet 4b28da769e1f46e1b05bd25e8428316f (table test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]) successfully deleted
I20240501 14:10:26.897532 20503 raft_consensus.cc:1050] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192: attempting to promote NON_VOTER 5ce107fc401f4105be9947cf24c8c55e to VOTER
I20240501 14:10:26.898094 20503 consensus_queue.cc:237] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } }
I20240501 14:10:26.899375 19968 raft_consensus.cc:1261] T 352498cc20094372a957c030cebf375e P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 14:10:26.899652 20379 raft_consensus.cc:1261] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e [term 1 LEARNER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 14:10:26.899736 20502 consensus_queue.cc:1035] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:10:26.899652 19895 raft_consensus.cc:1261] T 352498cc20094372a957c030cebf375e P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 14:10:26.900314 20463 consensus_queue.cc:1035] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:10:26.900527 20503 consensus_queue.cc:1035] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:10:26.902688 20502 raft_consensus.cc:2932] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:26.904471 19896 raft_consensus.cc:2932] T 352498cc20094372a957c030cebf375e P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:26.904471 20379 raft_consensus.cc:2932] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:26.906284 20228 raft_consensus.cc:1203] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240501 14:10:26.906849 19720 catalog_manager.cc:5484] T 352498cc20094372a957c030cebf375e P c4d2a3d7c38d4794a1ee78eaef2940d1 reported cstate change: config changed from index 27 to 28, 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7d4b038952284c81a07423e42a349192" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:26.907123 19974 raft_consensus.cc:2932] T 352498cc20094372a957c030cebf375e P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:26.910171 19823 consensus_queue.cc:237] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } }
I20240501 14:10:26.911578 19974 raft_consensus.cc:1261] T 352498cc20094372a957c030cebf375e P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 14:10:26.911844 20379 raft_consensus.cc:1261] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e [term 1 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 14:10:26.912025 20503 consensus_queue.cc:1035] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:10:26.912869 20503 consensus_queue.cc:1035] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:10:26.913482 20502 raft_consensus.cc:2932] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:26.914166 19968 raft_consensus.cc:2932] T 352498cc20094372a957c030cebf375e P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:26.915148 19708 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 352498cc20094372a957c030cebf375e with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:10:26.916457 19714 catalog_manager.cc:5484] T 352498cc20094372a957c030cebf375e P 94100faea1434b27a8c6d129d1b562bc reported cstate change: config changed from index 28 to 29, VOTER c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194) evicted. New cstate: current_term: 1 leader_uuid: "7d4b038952284c81a07423e42a349192" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:26.917207 20379 raft_consensus.cc:2932] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:26.918757 19885 tablet_service.cc:1514] Processing DeleteTablet for tablet 352498cc20094372a957c030cebf375e with delete_type TABLET_DATA_TOMBSTONED (TS c4d2a3d7c38d4794a1ee78eaef2940d1 not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:56764
I20240501 14:10:26.919166 20508 tablet_replica.cc:330] T 352498cc20094372a957c030cebf375e P c4d2a3d7c38d4794a1ee78eaef2940d1: stopping tablet replica
I20240501 14:10:26.919308 20508 raft_consensus.cc:2229] T 352498cc20094372a957c030cebf375e P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:26.919461 20508 raft_consensus.cc:2258] T 352498cc20094372a957c030cebf375e P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:26.920133 20508 ts_tablet_manager.cc:1875] T 352498cc20094372a957c030cebf375e P c4d2a3d7c38d4794a1ee78eaef2940d1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:26.922556 20508 ts_tablet_manager.cc:1888] T 352498cc20094372a957c030cebf375e P c4d2a3d7c38d4794a1ee78eaef2940d1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20240501 14:10:26.922638 20508 log.cc:1196] T 352498cc20094372a957c030cebf375e P c4d2a3d7c38d4794a1ee78eaef2940d1: Deleting WAL directory at /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-1-root/wals/352498cc20094372a957c030cebf375e
I20240501 14:10:26.923091 19706 catalog_manager.cc:4830] TS c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591): tablet 352498cc20094372a957c030cebf375e (table test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]) successfully deleted
I20240501 14:10:27.134654 20496 raft_consensus.cc:1050] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1: attempting to promote NON_VOTER 5ce107fc401f4105be9947cf24c8c55e to VOTER
I20240501 14:10:27.135092 20496 consensus_queue.cc:237] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } }
I20240501 14:10:27.136440 19819 raft_consensus.cc:1261] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 14:10:27.136440 19974 raft_consensus.cc:1261] T 515893894ce0478caa58378e6c2e047b P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 14:10:27.136641 20379 raft_consensus.cc:1261] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e [term 1 LEARNER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 14:10:27.136857 20466 consensus_queue.cc:1035] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:10:27.137015 20492 consensus_queue.cc:1035] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:10:27.137187 20489 consensus_queue.cc:1035] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:10:27.138763 20492 raft_consensus.cc:2932] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:27.139195 19974 raft_consensus.cc:2932] T 515893894ce0478caa58378e6c2e047b P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:27.140170 19819 raft_consensus.cc:2932] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:27.141583 20379 raft_consensus.cc:2932] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:27.142031 19714 catalog_manager.cc:5484] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 reported cstate change: config changed from index 27 to 28, 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:10:27.145108 19896 consensus_queue.cc:237] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } }
I20240501 14:10:27.146199 20379 raft_consensus.cc:1261] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e [term 1 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 14:10:27.146317 19819 raft_consensus.cc:1261] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 14:10:27.146624 20492 consensus_queue.cc:1035] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:10:27.146904 20466 consensus_queue.cc:1035] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:10:27.148144 20492 raft_consensus.cc:2932] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 94100faea1434b27a8c6d129d1b562bc (127.16.13.195) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:27.148522 20379 raft_consensus.cc:2932] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 94100faea1434b27a8c6d129d1b562bc (127.16.13.195) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:27.148953 19819 raft_consensus.cc:2932] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 94100faea1434b27a8c6d129d1b562bc (127.16.13.195) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:27.149657 19706 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 515893894ce0478caa58378e6c2e047b with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:10:27.150661 19720 catalog_manager.cc:5484] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e reported cstate change: config changed from index 28 to 29, VOTER 94100faea1434b27a8c6d129d1b562bc (127.16.13.195) evicted. New cstate: current_term: 1 leader_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:27.152966 19964 tablet_service.cc:1514] Processing DeleteTablet for tablet 515893894ce0478caa58378e6c2e047b with delete_type TABLET_DATA_TOMBSTONED (TS 94100faea1434b27a8c6d129d1b562bc not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:52786
I20240501 14:10:27.153345 20513 tablet_replica.cc:330] T 515893894ce0478caa58378e6c2e047b P 94100faea1434b27a8c6d129d1b562bc: stopping tablet replica
I20240501 14:10:27.153509 20513 raft_consensus.cc:2229] T 515893894ce0478caa58378e6c2e047b P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:27.153682 20513 raft_consensus.cc:2258] T 515893894ce0478caa58378e6c2e047b P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:27.154211 20513 ts_tablet_manager.cc:1875] T 515893894ce0478caa58378e6c2e047b P 94100faea1434b27a8c6d129d1b562bc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:27.155989 20513 ts_tablet_manager.cc:1888] T 515893894ce0478caa58378e6c2e047b P 94100faea1434b27a8c6d129d1b562bc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20240501 14:10:27.156060 20513 log.cc:1196] T 515893894ce0478caa58378e6c2e047b P 94100faea1434b27a8c6d129d1b562bc: Deleting WAL directory at /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-2-root/wals/515893894ce0478caa58378e6c2e047b
I20240501 14:10:27.156555 19708 catalog_manager.cc:4830] TS 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685): tablet 515893894ce0478caa58378e6c2e047b (table test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]) successfully deleted
I20240501 14:10:27.170589 20462 raft_consensus.cc:1050] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc: attempting to promote NON_VOTER fe34f459988040e291a748133a04015c to VOTER
I20240501 14:10:27.171020 20462 consensus_queue.cc:237] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } }
I20240501 14:10:27.172351 20228 raft_consensus.cc:1261] T 603909ab31c7462c9e2c893184a5fc6c P fe34f459988040e291a748133a04015c [term 1 LEARNER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 14:10:27.172817 19819 raft_consensus.cc:1261] T 603909ab31c7462c9e2c893184a5fc6c P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 14:10:27.173180 19895 raft_consensus.cc:1261] T 603909ab31c7462c9e2c893184a5fc6c P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 14:10:27.173338 20514 consensus_queue.cc:1035] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 14:10:27.173565 20515 consensus_queue.cc:1035] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 14:10:27.173840 20517 consensus_queue.cc:1035] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 14:10:27.175488 20514 raft_consensus.cc:2932] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, fe34f459988040e291a748133a04015c (127.16.13.196) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:27.176002 19895 raft_consensus.cc:2932] T 603909ab31c7462c9e2c893184a5fc6c P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, fe34f459988040e291a748133a04015c (127.16.13.196) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:27.177657 19819 raft_consensus.cc:2932] T 603909ab31c7462c9e2c893184a5fc6c P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, fe34f459988040e291a748133a04015c (127.16.13.196) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:27.178164 19718 catalog_manager.cc:5484] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc reported cstate change: config changed from index 26 to 27, fe34f459988040e291a748133a04015c (127.16.13.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "94100faea1434b27a8c6d129d1b562bc" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:10:27.179595 20228 raft_consensus.cc:2932] T 603909ab31c7462c9e2c893184a5fc6c P fe34f459988040e291a748133a04015c [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, fe34f459988040e291a748133a04015c (127.16.13.196) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:27.180737 19968 consensus_queue.cc:237] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } }
I20240501 14:10:27.181811 20228 raft_consensus.cc:1261] T 603909ab31c7462c9e2c893184a5fc6c P fe34f459988040e291a748133a04015c [term 1 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 14:10:27.181919 19823 raft_consensus.cc:1261] T 603909ab31c7462c9e2c893184a5fc6c P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 14:10:27.182641 20514 consensus_queue.cc:1035] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:10:27.182803 20517 consensus_queue.cc:1035] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:10:27.184180 20515 raft_consensus.cc:2932] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:27.184566 20228 raft_consensus.cc:2932] T 603909ab31c7462c9e2c893184a5fc6c P fe34f459988040e291a748133a04015c [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:27.184656 19823 raft_consensus.cc:2932] T 603909ab31c7462c9e2c893184a5fc6c P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:27.185868 19708 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 603909ab31c7462c9e2c893184a5fc6c with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:10:27.186692 19712 catalog_manager.cc:5484] T 603909ab31c7462c9e2c893184a5fc6c P fe34f459988040e291a748133a04015c reported cstate change: config changed from index 27 to 28, VOTER c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194) evicted. New cstate: current_term: 1 leader_uuid: "94100faea1434b27a8c6d129d1b562bc" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:27.189005 19885 tablet_service.cc:1514] Processing DeleteTablet for tablet 603909ab31c7462c9e2c893184a5fc6c with delete_type TABLET_DATA_TOMBSTONED (TS c4d2a3d7c38d4794a1ee78eaef2940d1 not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:56764
I20240501 14:10:27.189361 20508 tablet_replica.cc:330] T 603909ab31c7462c9e2c893184a5fc6c P c4d2a3d7c38d4794a1ee78eaef2940d1: stopping tablet replica
I20240501 14:10:27.189498 20508 raft_consensus.cc:2229] T 603909ab31c7462c9e2c893184a5fc6c P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:27.189631 20508 raft_consensus.cc:2258] T 603909ab31c7462c9e2c893184a5fc6c P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:27.189868 20514 raft_consensus.cc:1050] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc: attempting to promote NON_VOTER d0c0d1154175432ea3332f347685aa43 to VOTER
I20240501 14:10:27.190234 20514 consensus_queue.cc:237] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 15, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } }
I20240501 14:10:27.190495 20508 ts_tablet_manager.cc:1875] T 603909ab31c7462c9e2c893184a5fc6c P c4d2a3d7c38d4794a1ee78eaef2940d1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:27.191550 20304 raft_consensus.cc:1261] T a726c9cea76241df8a00d52d6175db7a P d0c0d1154175432ea3332f347685aa43 [term 2 LEARNER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240501 14:10:27.191802 19823 raft_consensus.cc:1261] T a726c9cea76241df8a00d52d6175db7a P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240501 14:10:27.191872 20514 consensus_queue.cc:1035] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 14:10:27.191969 19895 raft_consensus.cc:1261] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240501 14:10:27.192569 20515 consensus_queue.cc:1035] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 14:10:27.192772 20514 consensus_queue.cc:1035] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 14:10:27.193743 20508 ts_tablet_manager.cc:1888] T 603909ab31c7462c9e2c893184a5fc6c P c4d2a3d7c38d4794a1ee78eaef2940d1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20240501 14:10:27.193815 20508 log.cc:1196] T 603909ab31c7462c9e2c893184a5fc6c P c4d2a3d7c38d4794a1ee78eaef2940d1: Deleting WAL directory at /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-1-root/wals/603909ab31c7462c9e2c893184a5fc6c
I20240501 14:10:27.194361 19706 catalog_manager.cc:4830] TS c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591): tablet 603909ab31c7462c9e2c893184a5fc6c (table test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]) successfully deleted
I20240501 14:10:27.194605 20517 raft_consensus.cc:2932] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index 26 to 27, d0c0d1154175432ea3332f347685aa43 (127.16.13.197) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:27.194779 19895 raft_consensus.cc:2932] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index 26 to 27, d0c0d1154175432ea3332f347685aa43 (127.16.13.197) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:27.195909 19819 raft_consensus.cc:2932] T a726c9cea76241df8a00d52d6175db7a P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index 26 to 27, d0c0d1154175432ea3332f347685aa43 (127.16.13.197) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:27.196668 20304 raft_consensus.cc:2932] T a726c9cea76241df8a00d52d6175db7a P d0c0d1154175432ea3332f347685aa43 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index 26 to 27, d0c0d1154175432ea3332f347685aa43 (127.16.13.197) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:27.198194 19718 catalog_manager.cc:5484] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 reported cstate change: config changed from index 26 to 27, d0c0d1154175432ea3332f347685aa43 (127.16.13.197) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "94100faea1434b27a8c6d129d1b562bc" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:27.201566 19974 consensus_queue.cc:237] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 15, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } }
I20240501 14:10:27.202494 20304 raft_consensus.cc:1261] T a726c9cea76241df8a00d52d6175db7a P d0c0d1154175432ea3332f347685aa43 [term 2 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240501 14:10:27.202718 19895 raft_consensus.cc:1261] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240501 14:10:27.202924 20517 consensus_queue.cc:1035] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:10:27.203080 20514 consensus_queue.cc:1035] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:10:27.204691 20514 raft_consensus.cc:2932] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index 27 to 28, VOTER 7d4b038952284c81a07423e42a349192 (127.16.13.193) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:27.205073 20304 raft_consensus.cc:2932] T a726c9cea76241df8a00d52d6175db7a P d0c0d1154175432ea3332f347685aa43 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, VOTER 7d4b038952284c81a07423e42a349192 (127.16.13.193) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:27.205179 19895 raft_consensus.cc:2932] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, VOTER 7d4b038952284c81a07423e42a349192 (127.16.13.193) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:27.207003 19708 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a726c9cea76241df8a00d52d6175db7a with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:10:27.207191 19712 catalog_manager.cc:5484] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 reported cstate change: config changed from index 27 to 28, VOTER 7d4b038952284c81a07423e42a349192 (127.16.13.193) evicted. New cstate: current_term: 2 leader_uuid: "94100faea1434b27a8c6d129d1b562bc" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:27.209877 19806 tablet_service.cc:1514] Processing DeleteTablet for tablet a726c9cea76241df8a00d52d6175db7a with delete_type TABLET_DATA_TOMBSTONED (TS 7d4b038952284c81a07423e42a349192 not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:51456
I20240501 14:10:27.210098 20501 tablet_replica.cc:330] T a726c9cea76241df8a00d52d6175db7a P 7d4b038952284c81a07423e42a349192: stopping tablet replica
I20240501 14:10:27.210245 20501 raft_consensus.cc:2229] T a726c9cea76241df8a00d52d6175db7a P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:27.210377 20501 raft_consensus.cc:2258] T a726c9cea76241df8a00d52d6175db7a P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:27.211062 20501 ts_tablet_manager.cc:1875] T a726c9cea76241df8a00d52d6175db7a P 7d4b038952284c81a07423e42a349192: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:27.212859 20501 ts_tablet_manager.cc:1888] T a726c9cea76241df8a00d52d6175db7a P 7d4b038952284c81a07423e42a349192: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.27
I20240501 14:10:27.212939 20501 log.cc:1196] T a726c9cea76241df8a00d52d6175db7a P 7d4b038952284c81a07423e42a349192: Deleting WAL directory at /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-0-root/wals/a726c9cea76241df8a00d52d6175db7a
I20240501 14:10:27.213402 19708 catalog_manager.cc:4830] TS 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811): tablet a726c9cea76241df8a00d52d6175db7a (table test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]) successfully deleted
I20240501 14:10:27.425197 20525 raft_consensus.cc:1050] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192: attempting to promote NON_VOTER fe34f459988040e291a748133a04015c to VOTER
I20240501 14:10:27.425757 20525 consensus_queue.cc:237] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } }
I20240501 14:10:27.427136 20228 raft_consensus.cc:1261] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c [term 1 LEARNER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 14:10:27.427268 19895 raft_consensus.cc:1261] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 14:10:27.427268 19974 raft_consensus.cc:1261] T a47e39994ef3497e96bedd8a36133e00 P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 14:10:27.427704 20527 consensus_queue.cc:1035] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:10:27.427944 20525 consensus_queue.cc:1035] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:10:27.428149 20503 consensus_queue.cc:1035] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:10:27.430351 20525 raft_consensus.cc:2932] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, fe34f459988040e291a748133a04015c (127.16.13.196) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:27.430696 20228 raft_consensus.cc:2932] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, fe34f459988040e291a748133a04015c (127.16.13.196) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:27.430665 19974 raft_consensus.cc:2932] T a47e39994ef3497e96bedd8a36133e00 P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, fe34f459988040e291a748133a04015c (127.16.13.196) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:27.433027 19718 catalog_manager.cc:5484] T a47e39994ef3497e96bedd8a36133e00 P 94100faea1434b27a8c6d129d1b562bc reported cstate change: config changed from index 27 to 28, fe34f459988040e291a748133a04015c (127.16.13.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7d4b038952284c81a07423e42a349192" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:27.433085 19895 raft_consensus.cc:2932] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, fe34f459988040e291a748133a04015c (127.16.13.196) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:27.436817 19819 consensus_queue.cc:237] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } }
I20240501 14:10:27.437803 19895 raft_consensus.cc:1261] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 14:10:27.437803 20228 raft_consensus.cc:1261] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c [term 1 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 14:10:27.438141 20503 consensus_queue.cc:1035] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:10:27.438306 20527 consensus_queue.cc:1035] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:10:27.439826 20527 raft_consensus.cc:2932] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 94100faea1434b27a8c6d129d1b562bc (127.16.13.195) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:27.440130 20228 raft_consensus.cc:2932] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 94100faea1434b27a8c6d129d1b562bc (127.16.13.195) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:27.440241 19895 raft_consensus.cc:2932] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 94100faea1434b27a8c6d129d1b562bc (127.16.13.195) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:27.442165 19708 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a47e39994ef3497e96bedd8a36133e00 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:10:27.442248 19720 catalog_manager.cc:5484] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c reported cstate change: config changed from index 28 to 29, VOTER 94100faea1434b27a8c6d129d1b562bc (127.16.13.195) evicted. New cstate: current_term: 1 leader_uuid: "7d4b038952284c81a07423e42a349192" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:27.444423 19964 tablet_service.cc:1514] Processing DeleteTablet for tablet a47e39994ef3497e96bedd8a36133e00 with delete_type TABLET_DATA_TOMBSTONED (TS 94100faea1434b27a8c6d129d1b562bc not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:52786
I20240501 14:10:27.444625 20513 tablet_replica.cc:330] T a47e39994ef3497e96bedd8a36133e00 P 94100faea1434b27a8c6d129d1b562bc: stopping tablet replica
I20240501 14:10:27.444763 20513 raft_consensus.cc:2229] T a47e39994ef3497e96bedd8a36133e00 P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:27.444916 20513 raft_consensus.cc:2258] T a47e39994ef3497e96bedd8a36133e00 P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:27.445430 20513 ts_tablet_manager.cc:1875] T a47e39994ef3497e96bedd8a36133e00 P 94100faea1434b27a8c6d129d1b562bc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:27.447037 20513 ts_tablet_manager.cc:1888] T a47e39994ef3497e96bedd8a36133e00 P 94100faea1434b27a8c6d129d1b562bc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20240501 14:10:27.447100 20513 log.cc:1196] T a47e39994ef3497e96bedd8a36133e00 P 94100faea1434b27a8c6d129d1b562bc: Deleting WAL directory at /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-2-root/wals/a47e39994ef3497e96bedd8a36133e00
I20240501 14:10:27.447579 19708 catalog_manager.cc:4830] TS 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685): tablet a47e39994ef3497e96bedd8a36133e00 (table test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]) successfully deleted
I20240501 14:10:28.446820 19974 consensus_queue.cc:237] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } }
I20240501 14:10:28.448269 19823 raft_consensus.cc:1261] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 14:10:28.448602 19895 raft_consensus.cc:1261] T ffd3ce2e5a1a4a0182e578d471a022ae P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 14:10:28.448608 20514 consensus_queue.cc:1035] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 14:10:28.448989 20533 consensus_queue.cc:1035] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 14:10:28.450584 20534 raft_consensus.cc:2932] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } } }
I20240501 14:10:28.450932 19895 raft_consensus.cc:2932] T ffd3ce2e5a1a4a0182e578d471a022ae P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } } }
I20240501 14:10:28.452163 19823 raft_consensus.cc:2932] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } } }
I20240501 14:10:28.453158 19720 catalog_manager.cc:5484] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc reported cstate change: config changed from index -1 to 27, NON_VOTER 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) added. New cstate: current_term: 1 leader_uuid: "94100faea1434b27a8c6d129d1b562bc" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:28.455053 19895 consensus_queue.cc:237] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } }
I20240501 14:10:28.456617 19823 raft_consensus.cc:1261] T 57d657b87203496ca9d47d00fd98c1af P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 14:10:28.456969 19974 raft_consensus.cc:1261] T 57d657b87203496ca9d47d00fd98c1af P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 14:10:28.457589 20545 consensus_queue.cc:1035] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
W20240501 14:10:28.457821 19939 consensus_peers.cc:485] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc -> Peer 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198:46107): Couldn't send request to peer 5ce107fc401f4105be9947cf24c8c55e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffd3ce2e5a1a4a0182e578d471a022ae. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:28.457994 20466 consensus_queue.cc:1035] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 14:10:28.459614 20466 raft_consensus.cc:2932] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER fe34f459988040e291a748133a04015c (127.16.13.196) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } } }
I20240501 14:10:28.460085 19823 raft_consensus.cc:2932] T 57d657b87203496ca9d47d00fd98c1af P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER fe34f459988040e291a748133a04015c (127.16.13.196) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } } }
I20240501 14:10:28.462152 19712 catalog_manager.cc:5484] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 reported cstate change: config changed from index -1 to 27, NON_VOTER fe34f459988040e291a748133a04015c (127.16.13.196) added. New cstate: current_term: 1 leader_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:28.462205 19895 consensus_queue.cc:237] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 2.25, Last appended by leader: 24, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } }
W20240501 14:10:28.463407 19866 consensus_peers.cc:485] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 -> Peer fe34f459988040e291a748133a04015c (127.16.13.196:35779): Couldn't send request to peer fe34f459988040e291a748133a04015c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57d657b87203496ca9d47d00fd98c1af. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:28.463989 19974 raft_consensus.cc:1261] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 2 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240501 14:10:28.463999 19823 raft_consensus.cc:1261] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
W20240501 14:10:28.464175 19863 consensus_peers.cc:485] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 -> Peer d0c0d1154175432ea3332f347685aa43 (127.16.13.197:40045): Couldn't send request to peer d0c0d1154175432ea3332f347685aa43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f796e2ca050e4666b9eb2dfd8a9ee2ff. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:28.464336 19968 raft_consensus.cc:2932] T 57d657b87203496ca9d47d00fd98c1af P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER fe34f459988040e291a748133a04015c (127.16.13.196) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } } }
I20240501 14:10:28.464771 20551 consensus_queue.cc:1035] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 14:10:28.466179 20551 consensus_queue.cc:1035] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 14:10:28.466778 20544 raft_consensus.cc:2932] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 LEADER]: Committing config change with OpId 2.26: config changed from index -1 to 26, NON_VOTER d0c0d1154175432ea3332f347685aa43 (127.16.13.197) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } } }
I20240501 14:10:28.467182 19823 raft_consensus.cc:2932] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Committing config change with OpId 2.26: config changed from index -1 to 26, NON_VOTER d0c0d1154175432ea3332f347685aa43 (127.16.13.197) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } } }
I20240501 14:10:28.469640 19714 catalog_manager.cc:5484] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 reported cstate change: config changed from index -1 to 26, NON_VOTER d0c0d1154175432ea3332f347685aa43 (127.16.13.197) added. New cstate: current_term: 2 leader_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } } }
I20240501 14:10:28.469725 19968 consensus_queue.cc:237] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 17, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } }
I20240501 14:10:28.470247 19974 raft_consensus.cc:2932] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 2 FOLLOWER]: Committing config change with OpId 2.26: config changed from index -1 to 26, NON_VOTER d0c0d1154175432ea3332f347685aa43 (127.16.13.197) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } } }
I20240501 14:10:28.471146 19895 raft_consensus.cc:1261] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240501 14:10:28.471817 19823 raft_consensus.cc:1261] T 9d0888af56f2427c9387c3682f211a58 P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240501 14:10:28.472347 20514 consensus_queue.cc:1035] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20240501 14:10:28.472838 19941 consensus_peers.cc:485] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc -> Peer fe34f459988040e291a748133a04015c (127.16.13.196:35779): Couldn't send request to peer fe34f459988040e291a748133a04015c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d0888af56f2427c9387c3682f211a58. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:28.473054 20514 consensus_queue.cc:1035] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:10:28.474774 20533 raft_consensus.cc:2932] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER fe34f459988040e291a748133a04015c (127.16.13.196) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } } }
I20240501 14:10:28.475205 19895 raft_consensus.cc:2932] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER fe34f459988040e291a748133a04015c (127.16.13.196) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } } }
I20240501 14:10:28.475214 19823 raft_consensus.cc:2932] T 9d0888af56f2427c9387c3682f211a58 P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER fe34f459988040e291a748133a04015c (127.16.13.196) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } } }
I20240501 14:10:28.477362 19712 catalog_manager.cc:5484] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc reported cstate change: config changed from index -1 to 28, NON_VOTER fe34f459988040e291a748133a04015c (127.16.13.196) added. New cstate: current_term: 2 leader_uuid: "94100faea1434b27a8c6d129d1b562bc" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:28.477550 19823 consensus_queue.cc:237] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } }
I20240501 14:10:28.478785 19895 raft_consensus.cc:1261] T 2e6457de95c64fb19ae1d294c436c8f7 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 14:10:28.479007 19974 raft_consensus.cc:1261] T 2e6457de95c64fb19ae1d294c436c8f7 P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
W20240501 14:10:28.479056 19788 consensus_peers.cc:485] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 -> Peer 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198:46107): Couldn't send request to peer 5ce107fc401f4105be9947cf24c8c55e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e6457de95c64fb19ae1d294c436c8f7. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:28.479385 20531 consensus_queue.cc:1035] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 14:10:28.479550 20503 consensus_queue.cc:1035] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 14:10:28.481521 20503 raft_consensus.cc:2932] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } } }
I20240501 14:10:28.481915 19974 raft_consensus.cc:2932] T 2e6457de95c64fb19ae1d294c436c8f7 P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } } }
I20240501 14:10:28.483860 19895 raft_consensus.cc:2932] T 2e6457de95c64fb19ae1d294c436c8f7 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } } }
I20240501 14:10:28.484297 19823 consensus_queue.cc:237] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } }
I20240501 14:10:28.484727 19720 catalog_manager.cc:5484] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 reported cstate change: config changed from index -1 to 27, NON_VOTER 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) added. New cstate: current_term: 1 leader_uuid: "7d4b038952284c81a07423e42a349192" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:28.486274 20228 raft_consensus.cc:1261] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c [term 1 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 14:10:28.486672 19895 raft_consensus.cc:1261] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 14:10:28.487052 20560 consensus_queue.cc:1035] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:10:28.488879 20561 consensus_queue.cc:1035] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
W20240501 14:10:28.490470 19787 consensus_peers.cc:485] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 -> Peer d0c0d1154175432ea3332f347685aa43 (127.16.13.197:40045): Couldn't send request to peer d0c0d1154175432ea3332f347685aa43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a47e39994ef3497e96bedd8a36133e00. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:28.490861 20560 raft_consensus.cc:2932] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER d0c0d1154175432ea3332f347685aa43 (127.16.13.197) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } } }
I20240501 14:10:28.491303 19895 raft_consensus.cc:2932] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER d0c0d1154175432ea3332f347685aa43 (127.16.13.197) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } } }
I20240501 14:10:28.491497 20228 raft_consensus.cc:2932] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER d0c0d1154175432ea3332f347685aa43 (127.16.13.197) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } } }
I20240501 14:10:28.493634 19714 catalog_manager.cc:5484] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1 reported cstate change: config changed from index 29 to 30, NON_VOTER d0c0d1154175432ea3332f347685aa43 (127.16.13.197) added. New cstate: current_term: 1 leader_uuid: "7d4b038952284c81a07423e42a349192" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } } }
I20240501 14:10:28.805500 19775 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:10:28.807329 19823 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "352498cc20094372a957c030cebf375e"
dest_uuid: "7d4b038952284c81a07423e42a349192"
mode: GRACEFUL
new_leader_uuid: "5ce107fc401f4105be9947cf24c8c55e"
from {username='slave'} at 127.0.0.1:51516
I20240501 14:10:28.807505 19823 raft_consensus.cc:595] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [term 1 LEADER]: Received request to transfer leadership to TS 5ce107fc401f4105be9947cf24c8c55e
I20240501 14:10:28.808408 19895 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "515893894ce0478caa58378e6c2e047b"
dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
mode: GRACEFUL
new_leader_uuid: "7d4b038952284c81a07423e42a349192"
from {username='slave'} at 127.0.0.1:56822
I20240501 14:10:28.808521 19895 raft_consensus.cc:595] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Received request to transfer leadership to TS 7d4b038952284c81a07423e42a349192
I20240501 14:10:28.817833 19974 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a726c9cea76241df8a00d52d6175db7a"
dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
mode: GRACEFUL
new_leader_uuid: "d0c0d1154175432ea3332f347685aa43"
from {username='slave'} at 127.0.0.1:52850
I20240501 14:10:28.818001 19974 raft_consensus.cc:595] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [term 2 LEADER]: Received request to transfer leadership to TS d0c0d1154175432ea3332f347685aa43
I20240501 14:10:28.818763 19895 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b8922efb08b3491fa8d947793206bf7a"
dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
mode: GRACEFUL
new_leader_uuid: "94100faea1434b27a8c6d129d1b562bc"
from {username='slave'} at 127.0.0.1:56822
I20240501 14:10:28.818899 19895 raft_consensus.cc:595] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Received request to transfer leadership to TS 94100faea1434b27a8c6d129d1b562bc
I20240501 14:10:28.819515 19895 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f796e2ca050e4666b9eb2dfd8a9ee2ff"
dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
mode: GRACEFUL
new_leader_uuid: "94100faea1434b27a8c6d129d1b562bc"
from {username='slave'} at 127.0.0.1:56822
I20240501 14:10:28.819622 19895 raft_consensus.cc:595] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 LEADER]: Received request to transfer leadership to TS 94100faea1434b27a8c6d129d1b562bc
I20240501 14:10:28.820294 19974 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ffd3ce2e5a1a4a0182e578d471a022ae"
dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
mode: GRACEFUL
new_leader_uuid: "7d4b038952284c81a07423e42a349192"
from {username='slave'} at 127.0.0.1:52850
I20240501 14:10:28.820402 19974 raft_consensus.cc:595] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [term 1 LEADER]: Received request to transfer leadership to TS 7d4b038952284c81a07423e42a349192
I20240501 14:10:28.820650 19775 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 6
I20240501 14:10:28.820780 19775 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:10:28.946870 20551 raft_consensus.cc:979] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1: : Instructing follower 94100faea1434b27a8c6d129d1b562bc to start an election
I20240501 14:10:28.947059 20551 raft_consensus.cc:1067] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 LEADER]: Signalling peer 94100faea1434b27a8c6d129d1b562bc to start an election
I20240501 14:10:28.947506 19974 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f796e2ca050e4666b9eb2dfd8a9ee2ff"
dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
from {username='slave'} at 127.0.0.1:52796
I20240501 14:10:28.947650 19974 raft_consensus.cc:482] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:28.947721 19974 raft_consensus.cc:3037] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:10:28.948938 19974 raft_consensus.cc:504] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } }
I20240501 14:10:28.949389 19974 leader_election.cc:288] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [CANDIDATE]: Term 3 election: Requested vote from peers 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811), c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591)
I20240501 14:10:28.949714 19895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f796e2ca050e4666b9eb2dfd8a9ee2ff" candidate_uuid: "94100faea1434b27a8c6d129d1b562bc" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
I20240501 14:10:28.949724 19823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f796e2ca050e4666b9eb2dfd8a9ee2ff" candidate_uuid: "94100faea1434b27a8c6d129d1b562bc" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "7d4b038952284c81a07423e42a349192"
I20240501 14:10:28.949838 19895 raft_consensus.cc:3032] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:10:28.949887 19895 raft_consensus.cc:729] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c4d2a3d7c38d4794a1ee78eaef2940d1, State: Running, Role: LEADER
I20240501 14:10:28.949888 19823 raft_consensus.cc:3037] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:10:28.950037 19895 consensus_queue.cc:260] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } }
I20240501 14:10:28.950239 19895 raft_consensus.cc:3037] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:10:28.950922 19823 raft_consensus.cc:2454] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94100faea1434b27a8c6d129d1b562bc in term 3.
I20240501 14:10:28.951114 19895 raft_consensus.cc:2454] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94100faea1434b27a8c6d129d1b562bc in term 3.
I20240501 14:10:28.951201 19941 leader_election.cc:302] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4b038952284c81a07423e42a349192, 94100faea1434b27a8c6d129d1b562bc; no voters:
I20240501 14:10:28.951362 20533 raft_consensus.cc:2781] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:10:28.951449 20533 raft_consensus.cc:686] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 3 LEADER]: Becoming Leader. State: Replica: 94100faea1434b27a8c6d129d1b562bc, State: Running, Role: LEADER
I20240501 14:10:28.951581 20533 consensus_queue.cc:237] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } }
I20240501 14:10:28.953567 19720 catalog_manager.cc:5484] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc reported cstate change: term changed from 2 to 3, leader changed from c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194) to 94100faea1434b27a8c6d129d1b562bc (127.16.13.195). New cstate: current_term: 3 leader_uuid: "94100faea1434b27a8c6d129d1b562bc" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:28.956780 20560 raft_consensus.cc:979] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192: : Instructing follower 5ce107fc401f4105be9947cf24c8c55e to start an election
I20240501 14:10:28.956879 20560 raft_consensus.cc:1067] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [term 1 LEADER]: Signalling peer 5ce107fc401f4105be9947cf24c8c55e to start an election
I20240501 14:10:28.957226 20379 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "352498cc20094372a957c030cebf375e"
dest_uuid: "5ce107fc401f4105be9947cf24c8c55e"
from {username='slave'} at 127.0.0.1:49840
I20240501 14:10:28.957352 20379 raft_consensus.cc:482] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:28.957420 20379 raft_consensus.cc:3037] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:28.957645 19974 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f796e2ca050e4666b9eb2dfd8a9ee2ff"
dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52786
I20240501 14:10:28.957742 19974 raft_consensus.cc:595] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 3 LEADER]: Received request to transfer leadership
I20240501 14:10:28.958536 20379 raft_consensus.cc:504] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } }
I20240501 14:10:28.958956 20379 leader_election.cc:288] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e [CANDIDATE]: Term 2 election: Requested vote from peers 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811), 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:28.959602 19823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "352498cc20094372a957c030cebf375e" candidate_uuid: "5ce107fc401f4105be9947cf24c8c55e" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "7d4b038952284c81a07423e42a349192"
I20240501 14:10:28.959904 19823 raft_consensus.cc:3032] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:10:28.960027 19823 raft_consensus.cc:729] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7d4b038952284c81a07423e42a349192, State: Running, Role: LEADER
I20240501 14:10:28.960191 19823 consensus_queue.cc:260] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } }
I20240501 14:10:28.960397 19823 raft_consensus.cc:3037] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:28.961439 19823 raft_consensus.cc:2454] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce107fc401f4105be9947cf24c8c55e in term 2.
I20240501 14:10:28.961689 20345 leader_election.cc:302] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce107fc401f4105be9947cf24c8c55e, 7d4b038952284c81a07423e42a349192; no voters:
I20240501 14:10:28.961988 20573 raft_consensus.cc:2781] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:28.962208 20573 raft_consensus.cc:686] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e [term 2 LEADER]: Becoming Leader. State: Replica: 5ce107fc401f4105be9947cf24c8c55e, State: Running, Role: LEADER
I20240501 14:10:28.962415 20573 consensus_queue.cc:237] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } }
I20240501 14:10:28.963548 19968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "352498cc20094372a957c030cebf375e" candidate_uuid: "5ce107fc401f4105be9947cf24c8c55e" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
I20240501 14:10:28.963652 19968 raft_consensus.cc:3037] T 352498cc20094372a957c030cebf375e P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:28.964429 19974 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f796e2ca050e4666b9eb2dfd8a9ee2ff"
dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52786
I20240501 14:10:28.964524 19974 raft_consensus.cc:595] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 3 LEADER]: Received request to transfer leadership
I20240501 14:10:28.964561 19968 raft_consensus.cc:2454] T 352498cc20094372a957c030cebf375e P 94100faea1434b27a8c6d129d1b562bc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce107fc401f4105be9947cf24c8c55e in term 2.
W20240501 14:10:28.964792 19708 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for f796e2ca050e4666b9eb2dfd8a9ee2ff already in progress [suppressed 8 similar messages]
I20240501 14:10:28.965392 19720 catalog_manager.cc:5484] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e reported cstate change: term changed from 1 to 2, leader changed from 7d4b038952284c81a07423e42a349192 (127.16.13.193) to 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198). New cstate: current_term: 2 leader_uuid: "5ce107fc401f4105be9947cf24c8c55e" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:10:28.965549 19968 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f796e2ca050e4666b9eb2dfd8a9ee2ff"
dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52786
I20240501 14:10:28.965641 19968 raft_consensus.cc:595] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 3 LEADER]: Received request to transfer leadership
W20240501 14:10:28.965924 19769 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for f796e2ca050e4666b9eb2dfd8a9ee2ff already in progress
W20240501 14:10:28.965977 19769 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for f796e2ca050e4666b9eb2dfd8a9ee2ff already in progress
I20240501 14:10:28.976222 20577 ts_tablet_manager.cc:902] T 57d657b87203496ca9d47d00fd98c1af P fe34f459988040e291a748133a04015c: Initiating tablet copy from peer c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591)
I20240501 14:10:28.976872 20577 tablet_copy_client.cc:308] T 57d657b87203496ca9d47d00fd98c1af P fe34f459988040e291a748133a04015c: tablet copy: Beginning tablet copy session from remote peer at address 127.16.13.194:39591
I20240501 14:10:28.978745 20584 ts_tablet_manager.cc:902] T 2e6457de95c64fb19ae1d294c436c8f7 P 5ce107fc401f4105be9947cf24c8c55e: Initiating tablet copy from peer 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811)
I20240501 14:10:28.979396 20584 tablet_copy_client.cc:308] T 2e6457de95c64fb19ae1d294c436c8f7 P 5ce107fc401f4105be9947cf24c8c55e: tablet copy: Beginning tablet copy session from remote peer at address 127.16.13.193:38811
I20240501 14:10:28.979754 19834 tablet_copy_service.cc:140] P 7d4b038952284c81a07423e42a349192: Received BeginTabletCopySession request for tablet 2e6457de95c64fb19ae1d294c436c8f7 from peer 5ce107fc401f4105be9947cf24c8c55e ({username='slave'} at 127.0.0.1:51512)
I20240501 14:10:28.979851 19834 tablet_copy_service.cc:161] P 7d4b038952284c81a07423e42a349192: Beginning new tablet copy session on tablet 2e6457de95c64fb19ae1d294c436c8f7 from peer 5ce107fc401f4105be9947cf24c8c55e at {username='slave'} at 127.0.0.1:51512: session id = 5ce107fc401f4105be9947cf24c8c55e-2e6457de95c64fb19ae1d294c436c8f7
I20240501 14:10:28.980664 19834 tablet_copy_source_session.cc:205] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:28.981392 19910 tablet_copy_service.cc:140] P c4d2a3d7c38d4794a1ee78eaef2940d1: Received BeginTabletCopySession request for tablet 57d657b87203496ca9d47d00fd98c1af from peer fe34f459988040e291a748133a04015c ({username='slave'} at 127.0.0.1:56854)
I20240501 14:10:28.981508 20584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e6457de95c64fb19ae1d294c436c8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:28.981515 19910 tablet_copy_service.cc:161] P c4d2a3d7c38d4794a1ee78eaef2940d1: Beginning new tablet copy session on tablet 57d657b87203496ca9d47d00fd98c1af from peer fe34f459988040e291a748133a04015c at {username='slave'} at 127.0.0.1:56854: session id = fe34f459988040e291a748133a04015c-57d657b87203496ca9d47d00fd98c1af
I20240501 14:10:28.982476 19910 tablet_copy_source_session.cc:205] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:28.983215 20577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57d657b87203496ca9d47d00fd98c1af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:28.983966 20584 tablet_copy_client.cc:783] T 2e6457de95c64fb19ae1d294c436c8f7 P 5ce107fc401f4105be9947cf24c8c55e: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:28.984120 20584 tablet_copy_client.cc:647] T 2e6457de95c64fb19ae1d294c436c8f7 P 5ce107fc401f4105be9947cf24c8c55e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:28.985044 20588 tablet_copy_client.cc:1002] T 2e6457de95c64fb19ae1d294c436c8f7 P 5ce107fc401f4105be9947cf24c8c55e: tablet copy: Injecting latency into file download: 2000
I20240501 14:10:28.985103 20577 tablet_copy_client.cc:783] T 57d657b87203496ca9d47d00fd98c1af P fe34f459988040e291a748133a04015c: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:28.985230 20577 tablet_copy_client.cc:647] T 57d657b87203496ca9d47d00fd98c1af P fe34f459988040e291a748133a04015c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:28.986238 20581 tablet_copy_client.cc:1002] T 57d657b87203496ca9d47d00fd98c1af P fe34f459988040e291a748133a04015c: tablet copy: Injecting latency into file download: 2000
I20240501 14:10:29.015079 20514 raft_consensus.cc:979] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc: : Instructing follower 7d4b038952284c81a07423e42a349192 to start an election
I20240501 14:10:29.015202 20514 raft_consensus.cc:1067] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [term 1 LEADER]: Signalling peer 7d4b038952284c81a07423e42a349192 to start an election
I20240501 14:10:29.015563 19823 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ffd3ce2e5a1a4a0182e578d471a022ae"
dest_uuid: "7d4b038952284c81a07423e42a349192"
from {username='slave'} at 127.0.0.1:51478
I20240501 14:10:29.015702 19823 raft_consensus.cc:482] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:29.015771 19823 raft_consensus.cc:3037] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:29.016808 19823 raft_consensus.cc:504] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } }
I20240501 14:10:29.017148 19823 leader_election.cc:288] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [CANDIDATE]: Term 2 election: Requested vote from peers c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591), 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:29.017437 19896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd3ce2e5a1a4a0182e578d471a022ae" candidate_uuid: "7d4b038952284c81a07423e42a349192" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
I20240501 14:10:29.017434 19974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd3ce2e5a1a4a0182e578d471a022ae" candidate_uuid: "7d4b038952284c81a07423e42a349192" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
I20240501 14:10:29.017582 19896 raft_consensus.cc:3037] T ffd3ce2e5a1a4a0182e578d471a022ae P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:29.017583 19974 raft_consensus.cc:3032] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:10:29.017722 19974 raft_consensus.cc:729] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 94100faea1434b27a8c6d129d1b562bc, State: Running, Role: LEADER
I20240501 14:10:29.017876 19974 consensus_queue.cc:260] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } }
I20240501 14:10:29.018044 19974 raft_consensus.cc:3037] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:29.018690 19896 raft_consensus.cc:2454] T ffd3ce2e5a1a4a0182e578d471a022ae P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d4b038952284c81a07423e42a349192 in term 2.
I20240501 14:10:29.018975 19974 raft_consensus.cc:2454] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d4b038952284c81a07423e42a349192 in term 2.
I20240501 14:10:29.019030 19788 leader_election.cc:302] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4b038952284c81a07423e42a349192, c4d2a3d7c38d4794a1ee78eaef2940d1; no voters:
I20240501 14:10:29.019238 20560 raft_consensus.cc:2781] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:29.019344 20560 raft_consensus.cc:686] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [term 2 LEADER]: Becoming Leader. State: Replica: 7d4b038952284c81a07423e42a349192, State: Running, Role: LEADER
I20240501 14:10:29.019479 20560 consensus_queue.cc:237] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } }
I20240501 14:10:29.020974 19720 catalog_manager.cc:5484] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 reported cstate change: term changed from 1 to 2, leader changed from 94100faea1434b27a8c6d129d1b562bc (127.16.13.195) to 7d4b038952284c81a07423e42a349192 (127.16.13.193). New cstate: current_term: 2 leader_uuid: "7d4b038952284c81a07423e42a349192" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:29.061312 20589 ts_tablet_manager.cc:902] T 9d0888af56f2427c9387c3682f211a58 P fe34f459988040e291a748133a04015c: Initiating tablet copy from peer 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:29.061882 20589 tablet_copy_client.cc:308] T 9d0888af56f2427c9387c3682f211a58 P fe34f459988040e291a748133a04015c: tablet copy: Beginning tablet copy session from remote peer at address 127.16.13.195:36685
I20240501 14:10:29.062322 19985 tablet_copy_service.cc:140] P 94100faea1434b27a8c6d129d1b562bc: Received BeginTabletCopySession request for tablet 9d0888af56f2427c9387c3682f211a58 from peer fe34f459988040e291a748133a04015c ({username='slave'} at 127.0.0.1:52822)
I20240501 14:10:29.062427 19985 tablet_copy_service.cc:161] P 94100faea1434b27a8c6d129d1b562bc: Beginning new tablet copy session on tablet 9d0888af56f2427c9387c3682f211a58 from peer fe34f459988040e291a748133a04015c at {username='slave'} at 127.0.0.1:52822: session id = fe34f459988040e291a748133a04015c-9d0888af56f2427c9387c3682f211a58
I20240501 14:10:29.063303 19985 tablet_copy_source_session.cc:205] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:29.064092 20589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d0888af56f2427c9387c3682f211a58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:29.066604 20589 tablet_copy_client.cc:783] T 9d0888af56f2427c9387c3682f211a58 P fe34f459988040e291a748133a04015c: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:29.066888 20589 tablet_copy_client.cc:647] T 9d0888af56f2427c9387c3682f211a58 P fe34f459988040e291a748133a04015c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:29.067839 20591 tablet_copy_client.cc:1002] T 9d0888af56f2427c9387c3682f211a58 P fe34f459988040e291a748133a04015c: tablet copy: Injecting latency into file download: 2000
I20240501 14:10:29.068759 20594 ts_tablet_manager.cc:902] T a47e39994ef3497e96bedd8a36133e00 P d0c0d1154175432ea3332f347685aa43: Initiating tablet copy from peer 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811)
I20240501 14:10:29.069360 20594 tablet_copy_client.cc:308] T a47e39994ef3497e96bedd8a36133e00 P d0c0d1154175432ea3332f347685aa43: tablet copy: Beginning tablet copy session from remote peer at address 127.16.13.193:38811
I20240501 14:10:29.073403 19834 tablet_copy_service.cc:140] P 7d4b038952284c81a07423e42a349192: Received BeginTabletCopySession request for tablet a47e39994ef3497e96bedd8a36133e00 from peer d0c0d1154175432ea3332f347685aa43 ({username='slave'} at 127.0.0.1:51518)
I20240501 14:10:29.073490 19834 tablet_copy_service.cc:161] P 7d4b038952284c81a07423e42a349192: Beginning new tablet copy session on tablet a47e39994ef3497e96bedd8a36133e00 from peer d0c0d1154175432ea3332f347685aa43 at {username='slave'} at 127.0.0.1:51518: session id = d0c0d1154175432ea3332f347685aa43-a47e39994ef3497e96bedd8a36133e00
I20240501 14:10:29.074259 19834 tablet_copy_source_session.cc:205] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:29.074965 20594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a47e39994ef3497e96bedd8a36133e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:29.077284 20594 tablet_copy_client.cc:783] T a47e39994ef3497e96bedd8a36133e00 P d0c0d1154175432ea3332f347685aa43: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:29.077440 20594 tablet_copy_client.cc:647] T a47e39994ef3497e96bedd8a36133e00 P d0c0d1154175432ea3332f347685aa43: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:29.078428 20598 tablet_copy_client.cc:1002] T a47e39994ef3497e96bedd8a36133e00 P d0c0d1154175432ea3332f347685aa43: tablet copy: Injecting latency into file download: 2000
I20240501 14:10:29.149133 20514 raft_consensus.cc:979] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc: : Instructing follower d0c0d1154175432ea3332f347685aa43 to start an election
I20240501 14:10:29.149252 20514 raft_consensus.cc:1067] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [term 2 LEADER]: Signalling peer d0c0d1154175432ea3332f347685aa43 to start an election
I20240501 14:10:29.149638 20304 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a726c9cea76241df8a00d52d6175db7a"
dest_uuid: "d0c0d1154175432ea3332f347685aa43"
from {username='slave'} at 127.0.0.1:59526
I20240501 14:10:29.149767 20304 raft_consensus.cc:482] T a726c9cea76241df8a00d52d6175db7a P d0c0d1154175432ea3332f347685aa43 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:29.149842 20304 raft_consensus.cc:3037] T a726c9cea76241df8a00d52d6175db7a P d0c0d1154175432ea3332f347685aa43 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:10:29.150738 20304 raft_consensus.cc:504] T a726c9cea76241df8a00d52d6175db7a P d0c0d1154175432ea3332f347685aa43 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } }
I20240501 14:10:29.151216 20304 leader_election.cc:288] T a726c9cea76241df8a00d52d6175db7a P d0c0d1154175432ea3332f347685aa43 [CANDIDATE]: Term 3 election: Requested vote from peers c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591), 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:29.151414 19896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a726c9cea76241df8a00d52d6175db7a" candidate_uuid: "d0c0d1154175432ea3332f347685aa43" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
I20240501 14:10:29.151526 19896 raft_consensus.cc:3037] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:10:29.152084 19974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a726c9cea76241df8a00d52d6175db7a" candidate_uuid: "d0c0d1154175432ea3332f347685aa43" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
I20240501 14:10:29.152199 19974 raft_consensus.cc:3032] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:10:29.152242 19974 raft_consensus.cc:729] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 94100faea1434b27a8c6d129d1b562bc, State: Running, Role: LEADER
I20240501 14:10:29.152381 19974 consensus_queue.cc:260] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } }
I20240501 14:10:29.152576 19896 raft_consensus.cc:2454] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0c0d1154175432ea3332f347685aa43 in term 3.
I20240501 14:10:29.152585 19974 raft_consensus.cc:3037] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:10:29.152884 20268 leader_election.cc:302] T a726c9cea76241df8a00d52d6175db7a P d0c0d1154175432ea3332f347685aa43 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4d2a3d7c38d4794a1ee78eaef2940d1, d0c0d1154175432ea3332f347685aa43; no voters:
I20240501 14:10:29.153239 20601 raft_consensus.cc:2781] T a726c9cea76241df8a00d52d6175db7a P d0c0d1154175432ea3332f347685aa43 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:10:29.153465 20601 raft_consensus.cc:686] T a726c9cea76241df8a00d52d6175db7a P d0c0d1154175432ea3332f347685aa43 [term 3 LEADER]: Becoming Leader. State: Replica: d0c0d1154175432ea3332f347685aa43, State: Running, Role: LEADER
I20240501 14:10:29.153658 19974 raft_consensus.cc:2454] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0c0d1154175432ea3332f347685aa43 in term 3.
I20240501 14:10:29.153656 20601 consensus_queue.cc:237] T a726c9cea76241df8a00d52d6175db7a P d0c0d1154175432ea3332f347685aa43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } }
I20240501 14:10:29.155476 19720 catalog_manager.cc:5484] T a726c9cea76241df8a00d52d6175db7a P d0c0d1154175432ea3332f347685aa43 reported cstate change: term changed from 2 to 3, leader changed from 94100faea1434b27a8c6d129d1b562bc (127.16.13.195) to d0c0d1154175432ea3332f347685aa43 (127.16.13.197). New cstate: current_term: 3 leader_uuid: "d0c0d1154175432ea3332f347685aa43" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:10:29.206846 20551 raft_consensus.cc:979] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1: : Instructing follower 94100faea1434b27a8c6d129d1b562bc to start an election
I20240501 14:10:29.206979 20551 raft_consensus.cc:1067] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Signalling peer 94100faea1434b27a8c6d129d1b562bc to start an election
I20240501 14:10:29.207485 19974 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8922efb08b3491fa8d947793206bf7a"
dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
from {username='slave'} at 127.0.0.1:52796
I20240501 14:10:29.207653 19974 raft_consensus.cc:482] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:29.207749 19974 raft_consensus.cc:3037] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:29.208931 19974 raft_consensus.cc:504] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:29.209372 19974 leader_election.cc:288] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [CANDIDATE]: Term 2 election: Requested vote from peers 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811), c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591)
I20240501 14:10:29.209605 19823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8922efb08b3491fa8d947793206bf7a" candidate_uuid: "94100faea1434b27a8c6d129d1b562bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "7d4b038952284c81a07423e42a349192"
I20240501 14:10:29.209743 19823 raft_consensus.cc:3037] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:29.209692 19896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8922efb08b3491fa8d947793206bf7a" candidate_uuid: "94100faea1434b27a8c6d129d1b562bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
I20240501 14:10:29.209859 19896 raft_consensus.cc:3032] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:10:29.209919 19896 raft_consensus.cc:729] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c4d2a3d7c38d4794a1ee78eaef2940d1, State: Running, Role: LEADER
I20240501 14:10:29.210060 19896 consensus_queue.cc:260] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:29.210217 19896 raft_consensus.cc:3037] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:29.210600 20544 raft_consensus.cc:979] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1: : Instructing follower 7d4b038952284c81a07423e42a349192 to start an election
I20240501 14:10:29.210703 20544 raft_consensus.cc:1067] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Signalling peer 7d4b038952284c81a07423e42a349192 to start an election
I20240501 14:10:29.210834 19823 raft_consensus.cc:2454] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94100faea1434b27a8c6d129d1b562bc in term 2.
I20240501 14:10:29.211097 19896 raft_consensus.cc:2454] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94100faea1434b27a8c6d129d1b562bc in term 2.
I20240501 14:10:29.211104 19941 leader_election.cc:302] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4b038952284c81a07423e42a349192, 94100faea1434b27a8c6d129d1b562bc; no voters:
I20240501 14:10:29.211166 19823 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "515893894ce0478caa58378e6c2e047b"
dest_uuid: "7d4b038952284c81a07423e42a349192"
from {username='slave'} at 127.0.0.1:51470
I20240501 14:10:29.211385 19823 raft_consensus.cc:482] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:29.211424 20514 raft_consensus.cc:2781] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:29.211548 20514 raft_consensus.cc:686] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [term 2 LEADER]: Becoming Leader. State: Replica: 94100faea1434b27a8c6d129d1b562bc, State: Running, Role: LEADER
I20240501 14:10:29.211429 19823 raft_consensus.cc:3037] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:29.211704 20514 consensus_queue.cc:237] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:29.212608 19823 raft_consensus.cc:504] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } }
I20240501 14:10:29.212962 19823 leader_election.cc:288] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [CANDIDATE]: Term 2 election: Requested vote from peers c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591), 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198:46107)
I20240501 14:10:29.213222 19896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515893894ce0478caa58378e6c2e047b" candidate_uuid: "7d4b038952284c81a07423e42a349192" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
I20240501 14:10:29.213301 19896 raft_consensus.cc:3032] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:10:29.213327 19896 raft_consensus.cc:729] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c4d2a3d7c38d4794a1ee78eaef2940d1, State: Running, Role: LEADER
I20240501 14:10:29.213382 20379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515893894ce0478caa58378e6c2e047b" candidate_uuid: "7d4b038952284c81a07423e42a349192" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "5ce107fc401f4105be9947cf24c8c55e"
I20240501 14:10:29.213429 19896 consensus_queue.cc:260] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } }
I20240501 14:10:29.213526 20379 raft_consensus.cc:3037] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:29.213603 19896 raft_consensus.cc:3037] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:29.213924 19720 catalog_manager.cc:5484] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc reported cstate change: term changed from 1 to 2, leader changed from c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194) to 94100faea1434b27a8c6d129d1b562bc (127.16.13.195). New cstate: current_term: 2 leader_uuid: "94100faea1434b27a8c6d129d1b562bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: HEALTHY } } }
I20240501 14:10:29.214268 19896 raft_consensus.cc:2454] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d4b038952284c81a07423e42a349192 in term 2.
I20240501 14:10:29.214534 20379 raft_consensus.cc:2454] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d4b038952284c81a07423e42a349192 in term 2.
I20240501 14:10:29.214855 19788 leader_election.cc:302] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce107fc401f4105be9947cf24c8c55e, 7d4b038952284c81a07423e42a349192; no voters:
I20240501 14:10:29.215042 20560 raft_consensus.cc:2781] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:29.215173 20560 raft_consensus.cc:686] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [term 2 LEADER]: Becoming Leader. State: Replica: 7d4b038952284c81a07423e42a349192, State: Running, Role: LEADER
I20240501 14:10:29.215302 20560 consensus_queue.cc:237] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } }
I20240501 14:10:29.216863 19720 catalog_manager.cc:5484] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 reported cstate change: term changed from 1 to 2, leader changed from c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194) to 7d4b038952284c81a07423e42a349192 (127.16.13.193). New cstate: current_term: 2 leader_uuid: "7d4b038952284c81a07423e42a349192" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:29.351168 19823 raft_consensus.cc:1261] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Refusing update from remote peer 5ce107fc401f4105be9947cf24c8c55e: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 14:10:29.351588 20573 consensus_queue.cc:1035] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:10:29.354287 19974 raft_consensus.cc:1261] T 352498cc20094372a957c030cebf375e P 94100faea1434b27a8c6d129d1b562bc [term 2 FOLLOWER]: Refusing update from remote peer 5ce107fc401f4105be9947cf24c8c55e: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 14:10:29.354668 20573 consensus_queue.cc:1035] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:10:29.420318 19895 raft_consensus.cc:1261] T ffd3ce2e5a1a4a0182e578d471a022ae P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240501 14:10:29.420819 20560 consensus_queue.cc:1035] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:10:29.422994 19974 raft_consensus.cc:1261] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [term 2 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240501 14:10:29.423384 20612 consensus_queue.cc:1035] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20240501 14:10:29.424435 19788 consensus_peers.cc:485] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 -> Peer 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198:46107): Couldn't send request to peer 5ce107fc401f4105be9947cf24c8c55e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffd3ce2e5a1a4a0182e578d471a022ae. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:29.531893 19819 raft_consensus.cc:1261] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 3 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240501 14:10:29.532371 20614 consensus_queue.cc:1035] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
W20240501 14:10:29.533864 19938 consensus_peers.cc:485] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc -> Peer d0c0d1154175432ea3332f347685aa43 (127.16.13.197:40045): Couldn't send request to peer d0c0d1154175432ea3332f347685aa43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f796e2ca050e4666b9eb2dfd8a9ee2ff. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:29.534581 20514 raft_consensus.cc:979] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc: : Instructing follower 7d4b038952284c81a07423e42a349192 to start an election
I20240501 14:10:29.534715 20533 raft_consensus.cc:1067] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 3 LEADER]: Signalling peer 7d4b038952284c81a07423e42a349192 to start an election
I20240501 14:10:29.535238 19819 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f796e2ca050e4666b9eb2dfd8a9ee2ff"
dest_uuid: "7d4b038952284c81a07423e42a349192"
from {username='slave'} at 127.0.0.1:51478
I20240501 14:10:29.535379 19819 raft_consensus.cc:482] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:29.535444 19819 raft_consensus.cc:3037] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:10:29.535879 19895 raft_consensus.cc:1261] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 3 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240501 14:10:29.536264 20514 consensus_queue.cc:1035] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 14:10:29.536643 19819 raft_consensus.cc:504] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } }
I20240501 14:10:29.537016 19819 leader_election.cc:288] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [CANDIDATE]: Term 4 election: Requested vote from peers c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591), 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:29.537210 19896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f796e2ca050e4666b9eb2dfd8a9ee2ff" candidate_uuid: "7d4b038952284c81a07423e42a349192" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: true dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
I20240501 14:10:29.537603 19974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f796e2ca050e4666b9eb2dfd8a9ee2ff" candidate_uuid: "7d4b038952284c81a07423e42a349192" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: true dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
I20240501 14:10:29.537693 19974 raft_consensus.cc:3032] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:10:29.537729 19974 raft_consensus.cc:729] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 94100faea1434b27a8c6d129d1b562bc, State: Running, Role: LEADER
I20240501 14:10:29.537864 19974 consensus_queue.cc:260] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 27, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } }
I20240501 14:10:29.538049 19974 raft_consensus.cc:3037] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:10:29.538921 19974 raft_consensus.cc:2454] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d4b038952284c81a07423e42a349192 in term 4.
I20240501 14:10:29.539235 19790 leader_election.cc:302] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d4b038952284c81a07423e42a349192, 94100faea1434b27a8c6d129d1b562bc; no voters: c4d2a3d7c38d4794a1ee78eaef2940d1
I20240501 14:10:29.539430 20612 raft_consensus.cc:2781] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:10:29.539527 20612 raft_consensus.cc:686] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 4 LEADER]: Becoming Leader. State: Replica: 7d4b038952284c81a07423e42a349192, State: Running, Role: LEADER
I20240501 14:10:29.539659 20612 consensus_queue.cc:237] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 27, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } }
I20240501 14:10:29.541144 19714 catalog_manager.cc:5484] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 reported cstate change: term changed from 3 to 4, leader changed from 94100faea1434b27a8c6d129d1b562bc (127.16.13.195) to 7d4b038952284c81a07423e42a349192 (127.16.13.193). New cstate: current_term: 4 leader_uuid: "7d4b038952284c81a07423e42a349192" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:29.595448 19895 raft_consensus.cc:1261] T a726c9cea76241df8a00d52d6175db7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 3 FOLLOWER]: Refusing update from remote peer d0c0d1154175432ea3332f347685aa43: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240501 14:10:29.595818 20601 consensus_queue.cc:1035] T a726c9cea76241df8a00d52d6175db7a P d0c0d1154175432ea3332f347685aa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:10:29.597429 19896 raft_consensus.cc:1261] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240501 14:10:29.597802 20616 consensus_queue.cc:1035] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 14:10:29.598613 19974 raft_consensus.cc:1261] T a726c9cea76241df8a00d52d6175db7a P 94100faea1434b27a8c6d129d1b562bc [term 3 FOLLOWER]: Refusing update from remote peer d0c0d1154175432ea3332f347685aa43: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240501 14:10:29.599148 20602 consensus_queue.cc:1035] T a726c9cea76241df8a00d52d6175db7a P d0c0d1154175432ea3332f347685aa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:10:29.601686 19823 raft_consensus.cc:1261] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240501 14:10:29.601981 20614 consensus_queue.cc:1035] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 14:10:29.717336 20379 raft_consensus.cc:1261] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e [term 2 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 14:10:29.717800 20612 consensus_queue.cc:1035] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:10:29.720216 19896 raft_consensus.cc:1261] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 14:10:29.720714 20612 consensus_queue.cc:1035] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:10:30.006673 19968 raft_consensus.cc:1261] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 4 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 4 index: 28. (index mismatch)
I20240501 14:10:30.007102 20562 consensus_queue.cc:1035] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20240501 14:10:30.009253 19787 consensus_peers.cc:485] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 -> Peer d0c0d1154175432ea3332f347685aa43 (127.16.13.197:40045): Couldn't send request to peer d0c0d1154175432ea3332f347685aa43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f796e2ca050e4666b9eb2dfd8a9ee2ff. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:30.009474 19896 raft_consensus.cc:3037] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:10:30.010737 19896 raft_consensus.cc:1261] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 4 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 4 index: 28. (index mismatch)
I20240501 14:10:30.011184 20562 consensus_queue.cc:1035] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:10:30.013907 20629 ts_tablet_manager.cc:902] T ffd3ce2e5a1a4a0182e578d471a022ae P 5ce107fc401f4105be9947cf24c8c55e: Initiating tablet copy from peer 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811)
I20240501 14:10:30.014521 20629 tablet_copy_client.cc:308] T ffd3ce2e5a1a4a0182e578d471a022ae P 5ce107fc401f4105be9947cf24c8c55e: tablet copy: Beginning tablet copy session from remote peer at address 127.16.13.193:38811
I20240501 14:10:30.015022 19834 tablet_copy_service.cc:140] P 7d4b038952284c81a07423e42a349192: Received BeginTabletCopySession request for tablet ffd3ce2e5a1a4a0182e578d471a022ae from peer 5ce107fc401f4105be9947cf24c8c55e ({username='slave'} at 127.0.0.1:51512)
I20240501 14:10:30.015118 19834 tablet_copy_service.cc:161] P 7d4b038952284c81a07423e42a349192: Beginning new tablet copy session on tablet ffd3ce2e5a1a4a0182e578d471a022ae from peer 5ce107fc401f4105be9947cf24c8c55e at {username='slave'} at 127.0.0.1:51512: session id = 5ce107fc401f4105be9947cf24c8c55e-ffd3ce2e5a1a4a0182e578d471a022ae
I20240501 14:10:30.016151 19834 tablet_copy_source_session.cc:205] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:30.016920 20629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffd3ce2e5a1a4a0182e578d471a022ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:30.019183 20629 tablet_copy_client.cc:783] T ffd3ce2e5a1a4a0182e578d471a022ae P 5ce107fc401f4105be9947cf24c8c55e: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:30.019349 20629 tablet_copy_client.cc:647] T ffd3ce2e5a1a4a0182e578d471a022ae P 5ce107fc401f4105be9947cf24c8c55e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:30.020267 20633 tablet_copy_client.cc:1002] T ffd3ce2e5a1a4a0182e578d471a022ae P 5ce107fc401f4105be9947cf24c8c55e: tablet copy: Injecting latency into file download: 2000
I20240501 14:10:30.504520 20634 ts_tablet_manager.cc:902] T f796e2ca050e4666b9eb2dfd8a9ee2ff P d0c0d1154175432ea3332f347685aa43: Initiating tablet copy from peer 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811)
I20240501 14:10:30.505128 20634 tablet_copy_client.cc:308] T f796e2ca050e4666b9eb2dfd8a9ee2ff P d0c0d1154175432ea3332f347685aa43: tablet copy: Beginning tablet copy session from remote peer at address 127.16.13.193:38811
I20240501 14:10:30.505550 19834 tablet_copy_service.cc:140] P 7d4b038952284c81a07423e42a349192: Received BeginTabletCopySession request for tablet f796e2ca050e4666b9eb2dfd8a9ee2ff from peer d0c0d1154175432ea3332f347685aa43 ({username='slave'} at 127.0.0.1:51518)
I20240501 14:10:30.505659 19834 tablet_copy_service.cc:161] P 7d4b038952284c81a07423e42a349192: Beginning new tablet copy session on tablet f796e2ca050e4666b9eb2dfd8a9ee2ff from peer d0c0d1154175432ea3332f347685aa43 at {username='slave'} at 127.0.0.1:51518: session id = d0c0d1154175432ea3332f347685aa43-f796e2ca050e4666b9eb2dfd8a9ee2ff
I20240501 14:10:30.506647 19834 tablet_copy_source_session.cc:205] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:30.507418 20634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f796e2ca050e4666b9eb2dfd8a9ee2ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:30.509582 20634 tablet_copy_client.cc:783] T f796e2ca050e4666b9eb2dfd8a9ee2ff P d0c0d1154175432ea3332f347685aa43: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:30.509778 20634 tablet_copy_client.cc:647] T f796e2ca050e4666b9eb2dfd8a9ee2ff P d0c0d1154175432ea3332f347685aa43: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:30.510841 20637 tablet_copy_client.cc:1002] T f796e2ca050e4666b9eb2dfd8a9ee2ff P d0c0d1154175432ea3332f347685aa43: tablet copy: Injecting latency into file download: 2000
I20240501 14:10:30.821092 19775 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:10:30.823148 19823 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "515893894ce0478caa58378e6c2e047b"
dest_uuid: "7d4b038952284c81a07423e42a349192"
mode: GRACEFUL
new_leader_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
from {username='slave'} at 127.0.0.1:51516
I20240501 14:10:30.823323 19823 raft_consensus.cc:595] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [term 2 LEADER]: Received request to transfer leadership to TS c4d2a3d7c38d4794a1ee78eaef2940d1
I20240501 14:10:30.824275 19819 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a47e39994ef3497e96bedd8a36133e00"
dest_uuid: "7d4b038952284c81a07423e42a349192"
mode: GRACEFUL
new_leader_uuid: "fe34f459988040e291a748133a04015c"
from {username='slave'} at 127.0.0.1:51516
I20240501 14:10:30.824426 19819 raft_consensus.cc:595] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [term 1 LEADER]: Received request to transfer leadership to TS fe34f459988040e291a748133a04015c
I20240501 14:10:30.825089 19968 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b8922efb08b3491fa8d947793206bf7a"
dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
mode: GRACEFUL
new_leader_uuid: "7d4b038952284c81a07423e42a349192"
from {username='slave'} at 127.0.0.1:52850
I20240501 14:10:30.825206 19968 raft_consensus.cc:595] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [term 2 LEADER]: Received request to transfer leadership to TS 7d4b038952284c81a07423e42a349192
I20240501 14:10:30.826184 19819 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f796e2ca050e4666b9eb2dfd8a9ee2ff"
dest_uuid: "7d4b038952284c81a07423e42a349192"
mode: GRACEFUL
new_leader_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
from {username='slave'} at 127.0.0.1:51516
I20240501 14:10:30.826308 19819 raft_consensus.cc:595] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 4 LEADER]: Received request to transfer leadership to TS c4d2a3d7c38d4794a1ee78eaef2940d1
I20240501 14:10:30.826586 19775 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240501 14:10:30.826730 19775 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:10:30.985352 20584 tablet_copy_client.cc:523] T 2e6457de95c64fb19ae1d294c436c8f7 P 5ce107fc401f4105be9947cf24c8c55e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:30.986501 20577 tablet_copy_client.cc:523] T 57d657b87203496ca9d47d00fd98c1af P fe34f459988040e291a748133a04015c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:30.988755 20584 tablet_bootstrap.cc:492] T 2e6457de95c64fb19ae1d294c436c8f7 P 5ce107fc401f4105be9947cf24c8c55e: Bootstrap starting.
I20240501 14:10:30.988899 20577 tablet_bootstrap.cc:492] T 57d657b87203496ca9d47d00fd98c1af P fe34f459988040e291a748133a04015c: Bootstrap starting.
I20240501 14:10:31.006260 20560 raft_consensus.cc:979] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192: : Instructing follower c4d2a3d7c38d4794a1ee78eaef2940d1 to start an election
I20240501 14:10:31.006392 20560 raft_consensus.cc:1067] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 4 LEADER]: Signalling peer c4d2a3d7c38d4794a1ee78eaef2940d1 to start an election
I20240501 14:10:31.006955 19896 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f796e2ca050e4666b9eb2dfd8a9ee2ff"
dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
from {username='slave'} at 127.0.0.1:56780
I20240501 14:10:31.007107 19896 raft_consensus.cc:482] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:31.007174 19896 raft_consensus.cc:3037] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:10:31.008343 19896 raft_consensus.cc:504] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } }
I20240501 14:10:31.008715 19896 leader_election.cc:288] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 5 election: Requested vote from peers 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811), 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:31.009011 19819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f796e2ca050e4666b9eb2dfd8a9ee2ff" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: true dest_uuid: "7d4b038952284c81a07423e42a349192"
I20240501 14:10:31.009115 19819 raft_consensus.cc:3032] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:10:31.009153 19819 raft_consensus.cc:729] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7d4b038952284c81a07423e42a349192, State: Running, Role: LEADER
I20240501 14:10:31.009214 19968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f796e2ca050e4666b9eb2dfd8a9ee2ff" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: true dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
I20240501 14:10:31.009310 19968 raft_consensus.cc:3037] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:10:31.009352 19819 consensus_queue.cc:260] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 4.28, Last appended by leader: 28, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } }
I20240501 14:10:31.009805 19819 raft_consensus.cc:3037] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:10:31.010438 19968 raft_consensus.cc:2454] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 5.
I20240501 14:10:31.010824 19819 raft_consensus.cc:2454] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 5.
I20240501 14:10:31.011018 19866 leader_election.cc:302] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94100faea1434b27a8c6d129d1b562bc, c4d2a3d7c38d4794a1ee78eaef2940d1; no voters:
I20240501 14:10:31.011297 20551 raft_consensus.cc:2781] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:10:31.011453 20551 raft_consensus.cc:686] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 5 LEADER]: Becoming Leader. State: Replica: c4d2a3d7c38d4794a1ee78eaef2940d1, State: Running, Role: LEADER
I20240501 14:10:31.011590 20551 consensus_queue.cc:237] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 4.28, Last appended by leader: 28, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } }
I20240501 14:10:31.011732 20577 tablet_bootstrap.cc:492] T 57d657b87203496ca9d47d00fd98c1af P fe34f459988040e291a748133a04015c: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=107 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:31.012030 20577 tablet_bootstrap.cc:492] T 57d657b87203496ca9d47d00fd98c1af P fe34f459988040e291a748133a04015c: Bootstrap complete.
I20240501 14:10:31.012192 20577 ts_tablet_manager.cc:1367] T 57d657b87203496ca9d47d00fd98c1af P fe34f459988040e291a748133a04015c: Time spent bootstrapping tablet: real 0.023s user 0.023s sys 0.000s
I20240501 14:10:31.012665 20577 raft_consensus.cc:348] T 57d657b87203496ca9d47d00fd98c1af P fe34f459988040e291a748133a04015c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } }
I20240501 14:10:31.012780 20577 raft_consensus.cc:729] T 57d657b87203496ca9d47d00fd98c1af P fe34f459988040e291a748133a04015c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fe34f459988040e291a748133a04015c, State: Initialized, Role: LEARNER
I20240501 14:10:31.012955 20577 consensus_queue.cc:260] T 57d657b87203496ca9d47d00fd98c1af P fe34f459988040e291a748133a04015c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } }
I20240501 14:10:31.013290 20577 ts_tablet_manager.cc:1398] T 57d657b87203496ca9d47d00fd98c1af P fe34f459988040e291a748133a04015c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:31.013469 19714 catalog_manager.cc:5484] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 reported cstate change: term changed from 4 to 5, leader changed from 7d4b038952284c81a07423e42a349192 (127.16.13.193) to c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194). New cstate: current_term: 5 leader_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:31.014315 19910 tablet_copy_service.cc:342] P c4d2a3d7c38d4794a1ee78eaef2940d1: Request end of tablet copy session fe34f459988040e291a748133a04015c-57d657b87203496ca9d47d00fd98c1af received from {username='slave'} at 127.0.0.1:56854
I20240501 14:10:31.014386 19910 tablet_copy_service.cc:433] P c4d2a3d7c38d4794a1ee78eaef2940d1: ending tablet copy session fe34f459988040e291a748133a04015c-57d657b87203496ca9d47d00fd98c1af on tablet 57d657b87203496ca9d47d00fd98c1af with peer fe34f459988040e291a748133a04015c
I20240501 14:10:31.014631 20584 tablet_bootstrap.cc:492] T 2e6457de95c64fb19ae1d294c436c8f7 P 5ce107fc401f4105be9947cf24c8c55e: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=114 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:31.014956 20584 tablet_bootstrap.cc:492] T 2e6457de95c64fb19ae1d294c436c8f7 P 5ce107fc401f4105be9947cf24c8c55e: Bootstrap complete.
I20240501 14:10:31.015183 20584 ts_tablet_manager.cc:1367] T 2e6457de95c64fb19ae1d294c436c8f7 P 5ce107fc401f4105be9947cf24c8c55e: Time spent bootstrapping tablet: real 0.026s user 0.022s sys 0.001s
I20240501 14:10:31.015636 20584 raft_consensus.cc:348] T 2e6457de95c64fb19ae1d294c436c8f7 P 5ce107fc401f4105be9947cf24c8c55e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } }
I20240501 14:10:31.015753 20584 raft_consensus.cc:729] T 2e6457de95c64fb19ae1d294c436c8f7 P 5ce107fc401f4105be9947cf24c8c55e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5ce107fc401f4105be9947cf24c8c55e, State: Initialized, Role: LEARNER
I20240501 14:10:31.015875 20584 consensus_queue.cc:260] T 2e6457de95c64fb19ae1d294c436c8f7 P 5ce107fc401f4105be9947cf24c8c55e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } }
I20240501 14:10:31.016259 20584 ts_tablet_manager.cc:1398] T 2e6457de95c64fb19ae1d294c436c8f7 P 5ce107fc401f4105be9947cf24c8c55e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:31.016757 19834 tablet_copy_service.cc:342] P 7d4b038952284c81a07423e42a349192: Request end of tablet copy session 5ce107fc401f4105be9947cf24c8c55e-2e6457de95c64fb19ae1d294c436c8f7 received from {username='slave'} at 127.0.0.1:51512
I20240501 14:10:31.016834 19834 tablet_copy_service.cc:433] P 7d4b038952284c81a07423e42a349192: ending tablet copy session 5ce107fc401f4105be9947cf24c8c55e-2e6457de95c64fb19ae1d294c436c8f7 on tablet 2e6457de95c64fb19ae1d294c436c8f7 with peer 5ce107fc401f4105be9947cf24c8c55e
I20240501 14:10:31.023619 20228 raft_consensus.cc:1203] T 57d657b87203496ca9d47d00fd98c1af P fe34f459988040e291a748133a04015c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240501 14:10:31.055421 20614 raft_consensus.cc:979] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc: : Instructing follower 7d4b038952284c81a07423e42a349192 to start an election
I20240501 14:10:31.055537 20614 raft_consensus.cc:1067] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [term 2 LEADER]: Signalling peer 7d4b038952284c81a07423e42a349192 to start an election
I20240501 14:10:31.056196 19819 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8922efb08b3491fa8d947793206bf7a"
dest_uuid: "7d4b038952284c81a07423e42a349192"
from {username='slave'} at 127.0.0.1:51478
I20240501 14:10:31.056365 19819 raft_consensus.cc:482] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:31.056470 19819 raft_consensus.cc:3037] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:10:31.057583 19819 raft_consensus.cc:504] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:31.057994 19819 leader_election.cc:288] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [CANDIDATE]: Term 3 election: Requested vote from peers c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591), 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:31.058300 19895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8922efb08b3491fa8d947793206bf7a" candidate_uuid: "7d4b038952284c81a07423e42a349192" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
I20240501 14:10:31.058420 19895 raft_consensus.cc:3037] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:10:31.059432 19895 raft_consensus.cc:2454] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d4b038952284c81a07423e42a349192 in term 3.
I20240501 14:10:31.059655 19968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8922efb08b3491fa8d947793206bf7a" candidate_uuid: "7d4b038952284c81a07423e42a349192" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
I20240501 14:10:31.059734 19788 leader_election.cc:302] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4b038952284c81a07423e42a349192, c4d2a3d7c38d4794a1ee78eaef2940d1; no voters:
I20240501 14:10:31.059769 19968 raft_consensus.cc:3032] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:10:31.059844 19968 raft_consensus.cc:729] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 94100faea1434b27a8c6d129d1b562bc, State: Running, Role: LEADER
I20240501 14:10:31.059948 20648 raft_consensus.cc:2781] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:10:31.060038 20648 raft_consensus.cc:686] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [term 3 LEADER]: Becoming Leader. State: Replica: 7d4b038952284c81a07423e42a349192, State: Running, Role: LEADER
I20240501 14:10:31.060057 19968 consensus_queue.cc:260] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:31.060214 19968 raft_consensus.cc:3037] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:10:31.061098 19968 raft_consensus.cc:2454] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d4b038952284c81a07423e42a349192 in term 3.
I20240501 14:10:31.060166 20648 consensus_queue.cc:237] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:31.062733 19720 catalog_manager.cc:5484] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 reported cstate change: term changed from 2 to 3, leader changed from 94100faea1434b27a8c6d129d1b562bc (127.16.13.195) to 7d4b038952284c81a07423e42a349192 (127.16.13.193). New cstate: current_term: 3 leader_uuid: "7d4b038952284c81a07423e42a349192" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:31.068138 20589 tablet_copy_client.cc:523] T 9d0888af56f2427c9387c3682f211a58 P fe34f459988040e291a748133a04015c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:31.069665 20589 tablet_bootstrap.cc:492] T 9d0888af56f2427c9387c3682f211a58 P fe34f459988040e291a748133a04015c: Bootstrap starting.
I20240501 14:10:31.078709 20594 tablet_copy_client.cc:523] T a47e39994ef3497e96bedd8a36133e00 P d0c0d1154175432ea3332f347685aa43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:31.080348 20594 tablet_bootstrap.cc:492] T a47e39994ef3497e96bedd8a36133e00 P d0c0d1154175432ea3332f347685aa43: Bootstrap starting.
I20240501 14:10:31.090071 20648 raft_consensus.cc:979] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192: : Instructing follower fe34f459988040e291a748133a04015c to start an election
I20240501 14:10:31.090198 20648 raft_consensus.cc:1067] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [term 1 LEADER]: Signalling peer fe34f459988040e291a748133a04015c to start an election
I20240501 14:10:31.090655 20228 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a47e39994ef3497e96bedd8a36133e00"
dest_uuid: "fe34f459988040e291a748133a04015c"
from {username='slave'} at 127.0.0.1:47180
I20240501 14:10:31.090799 20228 raft_consensus.cc:482] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:31.090871 20228 raft_consensus.cc:3037] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:31.091766 20228 raft_consensus.cc:504] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } }
I20240501 14:10:31.092224 20228 leader_election.cc:288] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c [CANDIDATE]: Term 2 election: Requested vote from peers 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811), c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591)
I20240501 14:10:31.092675 19895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47e39994ef3497e96bedd8a36133e00" candidate_uuid: "fe34f459988040e291a748133a04015c" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
I20240501 14:10:31.092713 19819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47e39994ef3497e96bedd8a36133e00" candidate_uuid: "fe34f459988040e291a748133a04015c" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "7d4b038952284c81a07423e42a349192"
I20240501 14:10:31.092777 19895 raft_consensus.cc:3037] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:31.092775 19819 raft_consensus.cc:3032] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:10:31.092866 19819 raft_consensus.cc:729] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7d4b038952284c81a07423e42a349192, State: Running, Role: LEADER
I20240501 14:10:31.093022 19819 consensus_queue.cc:260] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } }
I20240501 14:10:31.093416 19819 raft_consensus.cc:3037] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:31.093787 19895 raft_consensus.cc:2454] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe34f459988040e291a748133a04015c in term 2.
I20240501 14:10:31.094226 20192 leader_election.cc:302] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4d2a3d7c38d4794a1ee78eaef2940d1, fe34f459988040e291a748133a04015c; no voters:
I20240501 14:10:31.094476 19819 raft_consensus.cc:2454] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe34f459988040e291a748133a04015c in term 2.
I20240501 14:10:31.094696 20643 raft_consensus.cc:2781] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:31.094938 20643 raft_consensus.cc:686] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c [term 2 LEADER]: Becoming Leader. State: Replica: fe34f459988040e291a748133a04015c, State: Running, Role: LEADER
I20240501 14:10:31.095088 20643 consensus_queue.cc:237] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } }
I20240501 14:10:31.090871 20589 tablet_bootstrap.cc:492] T 9d0888af56f2427c9387c3682f211a58 P fe34f459988040e291a748133a04015c: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=111 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:31.096374 20589 tablet_bootstrap.cc:492] T 9d0888af56f2427c9387c3682f211a58 P fe34f459988040e291a748133a04015c: Bootstrap complete.
I20240501 14:10:31.096623 20589 ts_tablet_manager.cc:1367] T 9d0888af56f2427c9387c3682f211a58 P fe34f459988040e291a748133a04015c: Time spent bootstrapping tablet: real 0.027s user 0.023s sys 0.000s
I20240501 14:10:31.096743 19720 catalog_manager.cc:5484] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c reported cstate change: term changed from 1 to 2, leader changed from 7d4b038952284c81a07423e42a349192 (127.16.13.193) to fe34f459988040e291a748133a04015c (127.16.13.196). New cstate: current_term: 2 leader_uuid: "fe34f459988040e291a748133a04015c" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:31.097061 20589 raft_consensus.cc:348] T 9d0888af56f2427c9387c3682f211a58 P fe34f459988040e291a748133a04015c [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } }
I20240501 14:10:31.097155 20589 raft_consensus.cc:729] T 9d0888af56f2427c9387c3682f211a58 P fe34f459988040e291a748133a04015c [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: fe34f459988040e291a748133a04015c, State: Initialized, Role: LEARNER
I20240501 14:10:31.097275 20589 consensus_queue.cc:260] T 9d0888af56f2427c9387c3682f211a58 P fe34f459988040e291a748133a04015c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } }
I20240501 14:10:31.097682 20589 ts_tablet_manager.cc:1398] T 9d0888af56f2427c9387c3682f211a58 P fe34f459988040e291a748133a04015c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:31.098332 19985 tablet_copy_service.cc:342] P 94100faea1434b27a8c6d129d1b562bc: Request end of tablet copy session fe34f459988040e291a748133a04015c-9d0888af56f2427c9387c3682f211a58 received from {username='slave'} at 127.0.0.1:52822
I20240501 14:10:31.098407 19985 tablet_copy_service.cc:433] P 94100faea1434b27a8c6d129d1b562bc: ending tablet copy session fe34f459988040e291a748133a04015c-9d0888af56f2427c9387c3682f211a58 on tablet 9d0888af56f2427c9387c3682f211a58 with peer fe34f459988040e291a748133a04015c
I20240501 14:10:31.106045 20594 tablet_bootstrap.cc:492] T a47e39994ef3497e96bedd8a36133e00 P d0c0d1154175432ea3332f347685aa43: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=105 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:31.106314 20594 tablet_bootstrap.cc:492] T a47e39994ef3497e96bedd8a36133e00 P d0c0d1154175432ea3332f347685aa43: Bootstrap complete.
I20240501 14:10:31.106492 20594 ts_tablet_manager.cc:1367] T a47e39994ef3497e96bedd8a36133e00 P d0c0d1154175432ea3332f347685aa43: Time spent bootstrapping tablet: real 0.026s user 0.024s sys 0.001s
I20240501 14:10:31.106990 20594 raft_consensus.cc:348] T a47e39994ef3497e96bedd8a36133e00 P d0c0d1154175432ea3332f347685aa43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } }
I20240501 14:10:31.107126 20594 raft_consensus.cc:729] T a47e39994ef3497e96bedd8a36133e00 P d0c0d1154175432ea3332f347685aa43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d0c0d1154175432ea3332f347685aa43, State: Initialized, Role: LEARNER
I20240501 14:10:31.107218 20594 consensus_queue.cc:260] T a47e39994ef3497e96bedd8a36133e00 P d0c0d1154175432ea3332f347685aa43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } }
I20240501 14:10:31.107501 20594 ts_tablet_manager.cc:1398] T a47e39994ef3497e96bedd8a36133e00 P d0c0d1154175432ea3332f347685aa43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:31.108078 19834 tablet_copy_service.cc:342] P 7d4b038952284c81a07423e42a349192: Request end of tablet copy session d0c0d1154175432ea3332f347685aa43-a47e39994ef3497e96bedd8a36133e00 received from {username='slave'} at 127.0.0.1:51518
I20240501 14:10:31.108148 19834 tablet_copy_service.cc:433] P 7d4b038952284c81a07423e42a349192: ending tablet copy session d0c0d1154175432ea3332f347685aa43-a47e39994ef3497e96bedd8a36133e00 on tablet a47e39994ef3497e96bedd8a36133e00 with peer d0c0d1154175432ea3332f347685aa43
I20240501 14:10:31.228009 20228 raft_consensus.cc:1203] T 9d0888af56f2427c9387c3682f211a58 P fe34f459988040e291a748133a04015c [term 2 LEARNER]: Deduplicated request from leader. Original: 2.27->[2.28-2.28] Dedup: 2.28->[]
I20240501 14:10:31.352066 20641 raft_consensus.cc:979] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192: : Instructing follower c4d2a3d7c38d4794a1ee78eaef2940d1 to start an election
I20240501 14:10:31.352214 20641 raft_consensus.cc:1067] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [term 2 LEADER]: Signalling peer c4d2a3d7c38d4794a1ee78eaef2940d1 to start an election
I20240501 14:10:31.352648 19896 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "515893894ce0478caa58378e6c2e047b"
dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
from {username='slave'} at 127.0.0.1:56780
I20240501 14:10:31.352792 19896 raft_consensus.cc:482] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:31.352864 19896 raft_consensus.cc:3037] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:10:31.353982 19896 raft_consensus.cc:504] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } }
I20240501 14:10:31.354400 19896 leader_election.cc:288] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 3 election: Requested vote from peers 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811), 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198:46107)
I20240501 14:10:31.354661 20379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515893894ce0478caa58378e6c2e047b" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "5ce107fc401f4105be9947cf24c8c55e"
I20240501 14:10:31.354743 19819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515893894ce0478caa58378e6c2e047b" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "7d4b038952284c81a07423e42a349192"
I20240501 14:10:31.354867 19819 raft_consensus.cc:3032] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:10:31.354915 19819 raft_consensus.cc:729] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7d4b038952284c81a07423e42a349192, State: Running, Role: LEADER
I20240501 14:10:31.354779 20379 raft_consensus.cc:3037] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:10:31.355055 19819 consensus_queue.cc:260] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } }
I20240501 14:10:31.355227 19819 raft_consensus.cc:3037] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:10:31.355902 20379 raft_consensus.cc:2454] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 3.
I20240501 14:10:31.356249 19864 leader_election.cc:302] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce107fc401f4105be9947cf24c8c55e, c4d2a3d7c38d4794a1ee78eaef2940d1; no voters:
I20240501 14:10:31.356271 19819 raft_consensus.cc:2454] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 3.
I20240501 14:10:31.356465 20642 raft_consensus.cc:2781] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:10:31.356575 20642 raft_consensus.cc:686] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 3 LEADER]: Becoming Leader. State: Replica: c4d2a3d7c38d4794a1ee78eaef2940d1, State: Running, Role: LEADER
I20240501 14:10:31.356717 20642 consensus_queue.cc:237] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } }
I20240501 14:10:31.358189 19720 catalog_manager.cc:5484] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 reported cstate change: term changed from 2 to 3, leader changed from 7d4b038952284c81a07423e42a349192 (127.16.13.193) to c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194). New cstate: current_term: 3 leader_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:31.451735 20379 raft_consensus.cc:1203] T 2e6457de95c64fb19ae1d294c436c8f7 P 5ce107fc401f4105be9947cf24c8c55e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240501 14:10:31.455030 19968 raft_consensus.cc:1261] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 5 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 4 index: 28. Preceding OpId from leader: term: 5 index: 29. (index mismatch)
I20240501 14:10:31.455420 20642 consensus_queue.cc:1035] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
W20240501 14:10:31.458596 19863 consensus_peers.cc:485] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 -> Peer d0c0d1154175432ea3332f347685aa43 (127.16.13.197:40045): Couldn't send request to peer d0c0d1154175432ea3332f347685aa43. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:31.458971 19823 raft_consensus.cc:1261] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 5 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 4 index: 28. Preceding OpId from leader: term: 5 index: 29. (index mismatch)
I20240501 14:10:31.459388 20660 consensus_queue.cc:1035] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:10:31.505136 19896 raft_consensus.cc:1261] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Refusing update from remote peer fe34f459988040e291a748133a04015c: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 14:10:31.505543 20643 consensus_queue.cc:1035] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:10:31.507992 19823 raft_consensus.cc:1261] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Refusing update from remote peer fe34f459988040e291a748133a04015c: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 14:10:31.508333 20666 consensus_queue.cc:1035] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:10:31.513007 20304 raft_consensus.cc:3037] T a47e39994ef3497e96bedd8a36133e00 P d0c0d1154175432ea3332f347685aa43 [term 1 LEARNER]: Advancing to term 2
I20240501 14:10:31.514098 20304 raft_consensus.cc:1261] T a47e39994ef3497e96bedd8a36133e00 P d0c0d1154175432ea3332f347685aa43 [term 2 LEARNER]: Refusing update from remote peer fe34f459988040e291a748133a04015c: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 14:10:31.514442 20643 consensus_queue.cc:1035] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:10:31.514727 20642 raft_consensus.cc:1050] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1: attempting to promote NON_VOTER fe34f459988040e291a748133a04015c to VOTER
I20240501 14:10:31.515151 20642 consensus_queue.cc:237] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } }
I20240501 14:10:31.516275 20666 raft_consensus.cc:1050] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c: attempting to promote NON_VOTER d0c0d1154175432ea3332f347685aa43 to VOTER
I20240501 14:10:31.516652 20666 consensus_queue.cc:237] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 30, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } }
I20240501 14:10:31.517133 20228 raft_consensus.cc:1261] T 57d657b87203496ca9d47d00fd98c1af P fe34f459988040e291a748133a04015c [term 1 LEARNER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 14:10:31.517259 19823 raft_consensus.cc:1261] T 57d657b87203496ca9d47d00fd98c1af P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 14:10:31.517984 20304 raft_consensus.cc:1261] T a47e39994ef3497e96bedd8a36133e00 P d0c0d1154175432ea3332f347685aa43 [term 2 LEARNER]: Refusing update from remote peer fe34f459988040e291a748133a04015c: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 14:10:31.517977 20660 consensus_queue.cc:1035] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:10:31.518324 20660 consensus_queue.cc:1035] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:10:31.518541 19968 raft_consensus.cc:1261] T 57d657b87203496ca9d47d00fd98c1af P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 14:10:31.518708 20643 consensus_queue.cc:1035] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 14:10:31.518754 19896 raft_consensus.cc:1261] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Refusing update from remote peer fe34f459988040e291a748133a04015c: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 14:10:31.519148 20643 consensus_queue.cc:1035] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 14:10:31.519253 19823 raft_consensus.cc:1261] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Refusing update from remote peer fe34f459988040e291a748133a04015c: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 14:10:31.519747 20643 consensus_queue.cc:1035] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 14:10:31.520571 20551 consensus_queue.cc:1035] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:10:31.522106 20653 raft_consensus.cc:2932] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 30 to 32, d0c0d1154175432ea3332f347685aa43 (127.16.13.197) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:31.522400 20304 raft_consensus.cc:2932] T a47e39994ef3497e96bedd8a36133e00 P d0c0d1154175432ea3332f347685aa43 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, d0c0d1154175432ea3332f347685aa43 (127.16.13.197) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:31.524029 19823 raft_consensus.cc:2932] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, d0c0d1154175432ea3332f347685aa43 (127.16.13.197) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:31.524257 19896 raft_consensus.cc:2932] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, d0c0d1154175432ea3332f347685aa43 (127.16.13.197) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:31.525842 19720 catalog_manager.cc:5484] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c reported cstate change: config changed from index 30 to 32, d0c0d1154175432ea3332f347685aa43 (127.16.13.197) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "fe34f459988040e291a748133a04015c" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:10:31.526916 19968 raft_consensus.cc:2932] T 57d657b87203496ca9d47d00fd98c1af P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, fe34f459988040e291a748133a04015c (127.16.13.196) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:31.528564 20551 raft_consensus.cc:2932] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, fe34f459988040e291a748133a04015c (127.16.13.196) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:31.529382 19720 catalog_manager.cc:5484] T 57d657b87203496ca9d47d00fd98c1af P 94100faea1434b27a8c6d129d1b562bc reported cstate change: config changed from index 27 to 28, fe34f459988040e291a748133a04015c (127.16.13.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:31.530351 20228 raft_consensus.cc:2932] T 57d657b87203496ca9d47d00fd98c1af P fe34f459988040e291a748133a04015c [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, fe34f459988040e291a748133a04015c (127.16.13.196) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:31.531627 20227 consensus_queue.cc:237] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } }
I20240501 14:10:31.532748 20304 raft_consensus.cc:1261] T a47e39994ef3497e96bedd8a36133e00 P d0c0d1154175432ea3332f347685aa43 [term 2 FOLLOWER]: Refusing update from remote peer fe34f459988040e291a748133a04015c: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240501 14:10:31.533058 20674 consensus_queue.cc:1035] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 14:10:31.534826 19821 raft_consensus.cc:1261] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Refusing update from remote peer fe34f459988040e291a748133a04015c: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240501 14:10:31.534749 20653 raft_consensus.cc:2932] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 32 to 33, VOTER c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:31.535281 19968 raft_consensus.cc:1261] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [term 3 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240501 14:10:31.535352 20304 raft_consensus.cc:2932] T a47e39994ef3497e96bedd8a36133e00 P d0c0d1154175432ea3332f347685aa43 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, VOTER c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:31.535645 20560 consensus_queue.cc:1035] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:10:31.535389 19823 raft_consensus.cc:2932] T 57d657b87203496ca9d47d00fd98c1af P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, fe34f459988040e291a748133a04015c (127.16.13.196) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:31.536150 20653 consensus_queue.cc:1035] T a47e39994ef3497e96bedd8a36133e00 P fe34f459988040e291a748133a04015c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 14:10:31.536685 19768 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a47e39994ef3497e96bedd8a36133e00 with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:10:31.537462 19821 raft_consensus.cc:2932] T a47e39994ef3497e96bedd8a36133e00 P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, VOTER c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:31.539409 19720 catalog_manager.cc:5484] T a47e39994ef3497e96bedd8a36133e00 P d0c0d1154175432ea3332f347685aa43 reported cstate change: config changed from index 32 to 33, VOTER c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194) evicted. New cstate: current_term: 2 leader_uuid: "fe34f459988040e291a748133a04015c" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:31.539547 19896 consensus_queue.cc:237] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } }
I20240501 14:10:31.540388 19895 raft_consensus.cc:1261] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 3 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240501 14:10:31.540784 20648 consensus_queue.cc:1035] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:10:31.540792 19974 raft_consensus.cc:1261] T 57d657b87203496ca9d47d00fd98c1af P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 14:10:31.541375 20642 consensus_queue.cc:1035] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:10:31.541857 20227 raft_consensus.cc:1261] T 57d657b87203496ca9d47d00fd98c1af P fe34f459988040e291a748133a04015c [term 1 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 14:10:31.542393 20642 consensus_queue.cc:1035] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:10:31.543077 19885 tablet_service.cc:1514] Processing DeleteTablet for tablet a47e39994ef3497e96bedd8a36133e00 with delete_type TABLET_DATA_TOMBSTONED (TS c4d2a3d7c38d4794a1ee78eaef2940d1 not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:56764
I20240501 14:10:31.543474 20681 tablet_replica.cc:330] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1: stopping tablet replica
I20240501 14:10:31.545086 20642 raft_consensus.cc:2932] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 7d4b038952284c81a07423e42a349192 (127.16.13.193) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:31.546924 19706 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 57d657b87203496ca9d47d00fd98c1af with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:10:31.547537 19974 raft_consensus.cc:2932] T 57d657b87203496ca9d47d00fd98c1af P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 7d4b038952284c81a07423e42a349192 (127.16.13.193) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:31.547555 20227 raft_consensus.cc:2932] T 57d657b87203496ca9d47d00fd98c1af P fe34f459988040e291a748133a04015c [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 7d4b038952284c81a07423e42a349192 (127.16.13.193) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:31.544135 20681 raft_consensus.cc:2229] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:31.549271 20681 raft_consensus.cc:2258] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:31.550074 19720 catalog_manager.cc:5484] T 57d657b87203496ca9d47d00fd98c1af P fe34f459988040e291a748133a04015c reported cstate change: config changed from index 28 to 29, VOTER 7d4b038952284c81a07423e42a349192 (127.16.13.193) evicted. New cstate: current_term: 1 leader_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:31.550979 20681 ts_tablet_manager.cc:1875] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:31.552739 19806 tablet_service.cc:1514] Processing DeleteTablet for tablet 57d657b87203496ca9d47d00fd98c1af with delete_type TABLET_DATA_TOMBSTONED (TS 7d4b038952284c81a07423e42a349192 not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:51456
I20240501 14:10:31.553239 20682 tablet_replica.cc:330] T 57d657b87203496ca9d47d00fd98c1af P 7d4b038952284c81a07423e42a349192: stopping tablet replica
I20240501 14:10:31.553323 20681 ts_tablet_manager.cc:1888] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.32
I20240501 14:10:31.553442 20682 raft_consensus.cc:2229] T 57d657b87203496ca9d47d00fd98c1af P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:31.553531 20681 log.cc:1196] T a47e39994ef3497e96bedd8a36133e00 P c4d2a3d7c38d4794a1ee78eaef2940d1: Deleting WAL directory at /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-1-root/wals/a47e39994ef3497e96bedd8a36133e00
I20240501 14:10:31.554028 20682 raft_consensus.cc:2258] T 57d657b87203496ca9d47d00fd98c1af P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:31.554042 19706 catalog_manager.cc:4830] TS c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591): tablet a47e39994ef3497e96bedd8a36133e00 (table test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]) successfully deleted
I20240501 14:10:31.554646 20682 ts_tablet_manager.cc:1875] T 57d657b87203496ca9d47d00fd98c1af P 7d4b038952284c81a07423e42a349192: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:31.556813 20682 ts_tablet_manager.cc:1888] T 57d657b87203496ca9d47d00fd98c1af P 7d4b038952284c81a07423e42a349192: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20240501 14:10:31.556887 20682 log.cc:1196] T 57d657b87203496ca9d47d00fd98c1af P 7d4b038952284c81a07423e42a349192: Deleting WAL directory at /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-0-root/wals/57d657b87203496ca9d47d00fd98c1af
I20240501 14:10:31.557361 19708 catalog_manager.cc:4830] TS 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811): tablet 57d657b87203496ca9d47d00fd98c1af (table test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]) successfully deleted
I20240501 14:10:31.643319 20684 raft_consensus.cc:1050] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc: attempting to promote NON_VOTER fe34f459988040e291a748133a04015c to VOTER
I20240501 14:10:31.643935 20684 consensus_queue.cc:237] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 17, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } }
I20240501 14:10:31.645350 19896 raft_consensus.cc:1261] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 14:10:31.645361 20227 raft_consensus.cc:1261] T 9d0888af56f2427c9387c3682f211a58 P fe34f459988040e291a748133a04015c [term 2 LEARNER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 14:10:31.645751 20683 consensus_queue.cc:1035] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:10:31.645845 19821 raft_consensus.cc:1261] T 9d0888af56f2427c9387c3682f211a58 P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 14:10:31.646050 20686 consensus_queue.cc:1035] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:10:31.646399 20683 consensus_queue.cc:1035] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:10:31.648130 20684 raft_consensus.cc:2932] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 28 to 29, fe34f459988040e291a748133a04015c (127.16.13.196) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:31.648505 20227 raft_consensus.cc:2932] T 9d0888af56f2427c9387c3682f211a58 P fe34f459988040e291a748133a04015c [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, fe34f459988040e291a748133a04015c (127.16.13.196) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:31.649334 19821 raft_consensus.cc:2932] T 9d0888af56f2427c9387c3682f211a58 P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, fe34f459988040e291a748133a04015c (127.16.13.196) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:31.651585 19718 catalog_manager.cc:5484] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc reported cstate change: config changed from index 28 to 29, fe34f459988040e291a748133a04015c (127.16.13.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "94100faea1434b27a8c6d129d1b562bc" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:10:31.652299 19896 raft_consensus.cc:2932] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, fe34f459988040e291a748133a04015c (127.16.13.196) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:31.656360 19974 consensus_queue.cc:237] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 17, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } }
I20240501 14:10:31.657325 19896 raft_consensus.cc:1261] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 14:10:31.657426 20227 raft_consensus.cc:1261] T 9d0888af56f2427c9387c3682f211a58 P fe34f459988040e291a748133a04015c [term 2 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 14:10:31.657680 20649 consensus_queue.cc:1035] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:10:31.657884 20684 consensus_queue.cc:1035] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:10:31.659591 20684 raft_consensus.cc:2932] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 7d4b038952284c81a07423e42a349192 (127.16.13.193) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:31.659878 19896 raft_consensus.cc:2932] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 7d4b038952284c81a07423e42a349192 (127.16.13.193) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:31.659878 20227 raft_consensus.cc:2932] T 9d0888af56f2427c9387c3682f211a58 P fe34f459988040e291a748133a04015c [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 7d4b038952284c81a07423e42a349192 (127.16.13.193) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } }
I20240501 14:10:31.661180 19708 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9d0888af56f2427c9387c3682f211a58 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:10:31.662114 19712 catalog_manager.cc:5484] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc reported cstate change: config changed from index 29 to 30, VOTER 7d4b038952284c81a07423e42a349192 (127.16.13.193) evicted. New cstate: current_term: 2 leader_uuid: "94100faea1434b27a8c6d129d1b562bc" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:10:31.664781 19806 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d0888af56f2427c9387c3682f211a58 with delete_type TABLET_DATA_TOMBSTONED (TS 7d4b038952284c81a07423e42a349192 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:51456
I20240501 14:10:31.664981 20682 tablet_replica.cc:330] T 9d0888af56f2427c9387c3682f211a58 P 7d4b038952284c81a07423e42a349192: stopping tablet replica
I20240501 14:10:31.665117 20682 raft_consensus.cc:2229] T 9d0888af56f2427c9387c3682f211a58 P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:31.665303 20682 raft_consensus.cc:2258] T 9d0888af56f2427c9387c3682f211a58 P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:31.665891 20682 ts_tablet_manager.cc:1875] T 9d0888af56f2427c9387c3682f211a58 P 7d4b038952284c81a07423e42a349192: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:31.667815 20682 ts_tablet_manager.cc:1888] T 9d0888af56f2427c9387c3682f211a58 P 7d4b038952284c81a07423e42a349192: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.29
I20240501 14:10:31.667901 20682 log.cc:1196] T 9d0888af56f2427c9387c3682f211a58 P 7d4b038952284c81a07423e42a349192: Deleting WAL directory at /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-0-root/wals/9d0888af56f2427c9387c3682f211a58
I20240501 14:10:31.668386 19708 catalog_manager.cc:4830] TS 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811): tablet 9d0888af56f2427c9387c3682f211a58 (table test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]) successfully deleted
I20240501 14:10:31.869515 20379 raft_consensus.cc:1261] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e [term 3 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 14:10:31.869917 20660 consensus_queue.cc:1035] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:10:31.873034 19821 raft_consensus.cc:1261] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [term 3 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 14:10:31.874295 20660 consensus_queue.cc:1035] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:10:31.974311 20641 raft_consensus.cc:1050] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192: attempting to promote NON_VOTER 5ce107fc401f4105be9947cf24c8c55e to VOTER
I20240501 14:10:31.974805 20641 consensus_queue.cc:237] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } }
I20240501 14:10:31.976159 20379 raft_consensus.cc:1261] T 2e6457de95c64fb19ae1d294c436c8f7 P 5ce107fc401f4105be9947cf24c8c55e [term 1 LEARNER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 14:10:31.976159 19974 raft_consensus.cc:1261] T 2e6457de95c64fb19ae1d294c436c8f7 P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 14:10:31.976267 19896 raft_consensus.cc:1261] T 2e6457de95c64fb19ae1d294c436c8f7 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 14:10:31.976735 20560 consensus_queue.cc:1035] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:10:31.976914 20695 consensus_queue.cc:1035] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:10:31.977034 20648 consensus_queue.cc:1035] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:10:31.978937 20695 raft_consensus.cc:2932] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:31.979279 20379 raft_consensus.cc:2932] T 2e6457de95c64fb19ae1d294c436c8f7 P 5ce107fc401f4105be9947cf24c8c55e [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:31.979305 19896 raft_consensus.cc:2932] T 2e6457de95c64fb19ae1d294c436c8f7 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:31.981002 19974 raft_consensus.cc:2932] T 2e6457de95c64fb19ae1d294c436c8f7 P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:31.981106 19720 catalog_manager.cc:5484] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 reported cstate change: config changed from index 27 to 28, 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7d4b038952284c81a07423e42a349192" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:10:31.983904 19823 consensus_queue.cc:237] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } }
I20240501 14:10:31.985584 19896 raft_consensus.cc:1261] T 2e6457de95c64fb19ae1d294c436c8f7 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 14:10:31.985839 20379 raft_consensus.cc:1261] T 2e6457de95c64fb19ae1d294c436c8f7 P 5ce107fc401f4105be9947cf24c8c55e [term 1 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 14:10:31.985913 20641 consensus_queue.cc:1035] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:10:31.986210 20560 consensus_queue.cc:1035] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:10:31.987598 20641 raft_consensus.cc:2932] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 94100faea1434b27a8c6d129d1b562bc (127.16.13.195) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:31.987983 19896 raft_consensus.cc:2932] T 2e6457de95c64fb19ae1d294c436c8f7 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 94100faea1434b27a8c6d129d1b562bc (127.16.13.195) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:31.988687 20379 raft_consensus.cc:2932] T 2e6457de95c64fb19ae1d294c436c8f7 P 5ce107fc401f4105be9947cf24c8c55e [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 94100faea1434b27a8c6d129d1b562bc (127.16.13.195) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:31.989521 19708 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2e6457de95c64fb19ae1d294c436c8f7 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:10:31.989962 19714 catalog_manager.cc:5484] T 2e6457de95c64fb19ae1d294c436c8f7 P 7d4b038952284c81a07423e42a349192 reported cstate change: config changed from index 28 to 29, VOTER 94100faea1434b27a8c6d129d1b562bc (127.16.13.195) evicted. New cstate: current_term: 1 leader_uuid: "7d4b038952284c81a07423e42a349192" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:10:31.992101 19964 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e6457de95c64fb19ae1d294c436c8f7 with delete_type TABLET_DATA_TOMBSTONED (TS 94100faea1434b27a8c6d129d1b562bc not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:52786
I20240501 14:10:31.992511 20699 tablet_replica.cc:330] T 2e6457de95c64fb19ae1d294c436c8f7 P 94100faea1434b27a8c6d129d1b562bc: stopping tablet replica
I20240501 14:10:31.992743 20699 raft_consensus.cc:2229] T 2e6457de95c64fb19ae1d294c436c8f7 P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:31.992941 20699 raft_consensus.cc:2258] T 2e6457de95c64fb19ae1d294c436c8f7 P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:31.993595 20699 ts_tablet_manager.cc:1875] T 2e6457de95c64fb19ae1d294c436c8f7 P 94100faea1434b27a8c6d129d1b562bc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:31.995654 20699 ts_tablet_manager.cc:1888] T 2e6457de95c64fb19ae1d294c436c8f7 P 94100faea1434b27a8c6d129d1b562bc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20240501 14:10:31.995721 20699 log.cc:1196] T 2e6457de95c64fb19ae1d294c436c8f7 P 94100faea1434b27a8c6d129d1b562bc: Deleting WAL directory at /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-2-root/wals/2e6457de95c64fb19ae1d294c436c8f7
I20240501 14:10:31.996294 19708 catalog_manager.cc:4830] TS 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685): tablet 2e6457de95c64fb19ae1d294c436c8f7 (table test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]) successfully deleted
I20240501 14:10:32.020535 20629 tablet_copy_client.cc:523] T ffd3ce2e5a1a4a0182e578d471a022ae P 5ce107fc401f4105be9947cf24c8c55e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:32.022112 20629 tablet_bootstrap.cc:492] T ffd3ce2e5a1a4a0182e578d471a022ae P 5ce107fc401f4105be9947cf24c8c55e: Bootstrap starting.
I20240501 14:10:32.045133 20629 tablet_bootstrap.cc:492] T ffd3ce2e5a1a4a0182e578d471a022ae P 5ce107fc401f4105be9947cf24c8c55e: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=118 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:32.045464 20629 tablet_bootstrap.cc:492] T ffd3ce2e5a1a4a0182e578d471a022ae P 5ce107fc401f4105be9947cf24c8c55e: Bootstrap complete.
I20240501 14:10:32.045661 20629 ts_tablet_manager.cc:1367] T ffd3ce2e5a1a4a0182e578d471a022ae P 5ce107fc401f4105be9947cf24c8c55e: Time spent bootstrapping tablet: real 0.024s user 0.023s sys 0.000s
I20240501 14:10:32.046350 20629 raft_consensus.cc:348] T ffd3ce2e5a1a4a0182e578d471a022ae P 5ce107fc401f4105be9947cf24c8c55e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } }
I20240501 14:10:32.046483 20629 raft_consensus.cc:729] T ffd3ce2e5a1a4a0182e578d471a022ae P 5ce107fc401f4105be9947cf24c8c55e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 5ce107fc401f4105be9947cf24c8c55e, State: Initialized, Role: LEARNER
I20240501 14:10:32.046610 20629 consensus_queue.cc:260] T ffd3ce2e5a1a4a0182e578d471a022ae P 5ce107fc401f4105be9947cf24c8c55e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } }
I20240501 14:10:32.046890 20629 ts_tablet_manager.cc:1398] T ffd3ce2e5a1a4a0182e578d471a022ae P 5ce107fc401f4105be9947cf24c8c55e: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 14:10:32.047410 19834 tablet_copy_service.cc:342] P 7d4b038952284c81a07423e42a349192: Request end of tablet copy session 5ce107fc401f4105be9947cf24c8c55e-ffd3ce2e5a1a4a0182e578d471a022ae received from {username='slave'} at 127.0.0.1:51512
I20240501 14:10:32.047484 19834 tablet_copy_service.cc:433] P 7d4b038952284c81a07423e42a349192: ending tablet copy session 5ce107fc401f4105be9947cf24c8c55e-ffd3ce2e5a1a4a0182e578d471a022ae on tablet ffd3ce2e5a1a4a0182e578d471a022ae with peer 5ce107fc401f4105be9947cf24c8c55e
I20240501 14:10:32.087740 20379 raft_consensus.cc:1203] T ffd3ce2e5a1a4a0182e578d471a022ae P 5ce107fc401f4105be9947cf24c8c55e [term 2 LEARNER]: Deduplicated request from leader. Original: 1.27->[2.28-2.28] Dedup: 2.28->[]
I20240501 14:10:32.511128 20634 tablet_copy_client.cc:523] T f796e2ca050e4666b9eb2dfd8a9ee2ff P d0c0d1154175432ea3332f347685aa43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:32.512517 20634 tablet_bootstrap.cc:492] T f796e2ca050e4666b9eb2dfd8a9ee2ff P d0c0d1154175432ea3332f347685aa43: Bootstrap starting.
I20240501 14:10:32.531489 20634 tablet_bootstrap.cc:492] T f796e2ca050e4666b9eb2dfd8a9ee2ff P d0c0d1154175432ea3332f347685aa43: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=84 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:32.531810 20634 tablet_bootstrap.cc:492] T f796e2ca050e4666b9eb2dfd8a9ee2ff P d0c0d1154175432ea3332f347685aa43: Bootstrap complete.
I20240501 14:10:32.532094 20634 ts_tablet_manager.cc:1367] T f796e2ca050e4666b9eb2dfd8a9ee2ff P d0c0d1154175432ea3332f347685aa43: Time spent bootstrapping tablet: real 0.020s user 0.020s sys 0.000s
I20240501 14:10:32.532572 20634 raft_consensus.cc:348] T f796e2ca050e4666b9eb2dfd8a9ee2ff P d0c0d1154175432ea3332f347685aa43 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } }
I20240501 14:10:32.532740 20634 raft_consensus.cc:729] T f796e2ca050e4666b9eb2dfd8a9ee2ff P d0c0d1154175432ea3332f347685aa43 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: d0c0d1154175432ea3332f347685aa43, State: Initialized, Role: LEARNER
I20240501 14:10:32.532850 20634 consensus_queue.cc:260] T f796e2ca050e4666b9eb2dfd8a9ee2ff P d0c0d1154175432ea3332f347685aa43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 4.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } }
I20240501 14:10:32.533218 20634 ts_tablet_manager.cc:1398] T f796e2ca050e4666b9eb2dfd8a9ee2ff P d0c0d1154175432ea3332f347685aa43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:32.533643 19834 tablet_copy_service.cc:342] P 7d4b038952284c81a07423e42a349192: Request end of tablet copy session d0c0d1154175432ea3332f347685aa43-f796e2ca050e4666b9eb2dfd8a9ee2ff received from {username='slave'} at 127.0.0.1:51518
I20240501 14:10:32.533730 19834 tablet_copy_service.cc:433] P 7d4b038952284c81a07423e42a349192: ending tablet copy session d0c0d1154175432ea3332f347685aa43-f796e2ca050e4666b9eb2dfd8a9ee2ff on tablet f796e2ca050e4666b9eb2dfd8a9ee2ff with peer d0c0d1154175432ea3332f347685aa43
I20240501 14:10:32.562590 20695 raft_consensus.cc:1050] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192: attempting to promote NON_VOTER 5ce107fc401f4105be9947cf24c8c55e to VOTER
I20240501 14:10:32.562990 20695 consensus_queue.cc:237] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 27, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } }
I20240501 14:10:32.564162 20379 raft_consensus.cc:1261] T ffd3ce2e5a1a4a0182e578d471a022ae P 5ce107fc401f4105be9947cf24c8c55e [term 2 LEARNER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 14:10:32.564193 19974 raft_consensus.cc:1261] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [term 2 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 14:10:32.564476 19896 raft_consensus.cc:1261] T ffd3ce2e5a1a4a0182e578d471a022ae P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 14:10:32.564519 20705 consensus_queue.cc:1035] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:10:32.564731 20695 consensus_queue.cc:1035] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:10:32.564925 20648 consensus_queue.cc:1035] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:10:32.567095 20695 raft_consensus.cc:2932] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 27 to 29, 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:32.567461 20379 raft_consensus.cc:2932] T ffd3ce2e5a1a4a0182e578d471a022ae P 5ce107fc401f4105be9947cf24c8c55e [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 27 to 29, 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:32.568073 19896 raft_consensus.cc:2932] T ffd3ce2e5a1a4a0182e578d471a022ae P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 27 to 29, 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:32.568933 19974 raft_consensus.cc:2932] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 27 to 29, 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:32.569803 19712 catalog_manager.cc:5484] T ffd3ce2e5a1a4a0182e578d471a022ae P 5ce107fc401f4105be9947cf24c8c55e reported cstate change: config changed from index 27 to 29, 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "7d4b038952284c81a07423e42a349192" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:32.573784 19823 consensus_queue.cc:237] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } }
I20240501 14:10:32.574731 20379 raft_consensus.cc:1261] T ffd3ce2e5a1a4a0182e578d471a022ae P 5ce107fc401f4105be9947cf24c8c55e [term 2 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 14:10:32.574807 19974 raft_consensus.cc:1261] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [term 2 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 14:10:32.575107 20560 consensus_queue.cc:1035] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:10:32.575263 20648 consensus_queue.cc:1035] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:10:32.576710 20648 raft_consensus.cc:2932] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:32.577181 19974 raft_consensus.cc:2932] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:32.578227 20379 raft_consensus.cc:2932] T ffd3ce2e5a1a4a0182e578d471a022ae P 5ce107fc401f4105be9947cf24c8c55e [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } }
I20240501 14:10:32.578393 19708 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ffd3ce2e5a1a4a0182e578d471a022ae with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:10:32.579123 19718 catalog_manager.cc:5484] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 reported cstate change: config changed from index 29 to 30, VOTER c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194) evicted. New cstate: current_term: 2 leader_uuid: "7d4b038952284c81a07423e42a349192" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:10:32.581310 19885 tablet_service.cc:1514] Processing DeleteTablet for tablet ffd3ce2e5a1a4a0182e578d471a022ae with delete_type TABLET_DATA_TOMBSTONED (TS c4d2a3d7c38d4794a1ee78eaef2940d1 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:56764
I20240501 14:10:32.581634 20712 tablet_replica.cc:330] T ffd3ce2e5a1a4a0182e578d471a022ae P c4d2a3d7c38d4794a1ee78eaef2940d1: stopping tablet replica
I20240501 14:10:32.581804 20712 raft_consensus.cc:2229] T ffd3ce2e5a1a4a0182e578d471a022ae P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:32.581990 20712 raft_consensus.cc:2258] T ffd3ce2e5a1a4a0182e578d471a022ae P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:32.582480 20712 ts_tablet_manager.cc:1875] T ffd3ce2e5a1a4a0182e578d471a022ae P c4d2a3d7c38d4794a1ee78eaef2940d1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:32.584385 20712 ts_tablet_manager.cc:1888] T ffd3ce2e5a1a4a0182e578d471a022ae P c4d2a3d7c38d4794a1ee78eaef2940d1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.29
I20240501 14:10:32.584463 20712 log.cc:1196] T ffd3ce2e5a1a4a0182e578d471a022ae P c4d2a3d7c38d4794a1ee78eaef2940d1: Deleting WAL directory at /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-1-root/wals/ffd3ce2e5a1a4a0182e578d471a022ae
I20240501 14:10:32.584961 19706 catalog_manager.cc:4830] TS c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591): tablet ffd3ce2e5a1a4a0182e578d471a022ae (table test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]) successfully deleted
I20240501 14:10:32.827061 19775 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:10:32.828948 19896 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "515893894ce0478caa58378e6c2e047b"
dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
mode: GRACEFUL
new_leader_uuid: "5ce107fc401f4105be9947cf24c8c55e"
from {username='slave'} at 127.0.0.1:56822
I20240501 14:10:32.829147 19896 raft_consensus.cc:595] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 3 LEADER]: Received request to transfer leadership to TS 5ce107fc401f4105be9947cf24c8c55e
I20240501 14:10:32.830034 19896 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f796e2ca050e4666b9eb2dfd8a9ee2ff"
dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
mode: GRACEFUL
new_leader_uuid: "7d4b038952284c81a07423e42a349192"
from {username='slave'} at 127.0.0.1:56822
I20240501 14:10:32.830161 19896 raft_consensus.cc:595] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 5 LEADER]: Received request to transfer leadership to TS 7d4b038952284c81a07423e42a349192
I20240501 14:10:32.830854 19821 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ffd3ce2e5a1a4a0182e578d471a022ae"
dest_uuid: "7d4b038952284c81a07423e42a349192"
mode: GRACEFUL
new_leader_uuid: "94100faea1434b27a8c6d129d1b562bc"
from {username='slave'} at 127.0.0.1:51516
I20240501 14:10:32.830945 19821 raft_consensus.cc:595] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [term 2 LEADER]: Received request to transfer leadership to TS 94100faea1434b27a8c6d129d1b562bc
I20240501 14:10:32.831202 19775 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240501 14:10:32.831331 19775 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:10:33.010124 20304 raft_consensus.cc:3037] T f796e2ca050e4666b9eb2dfd8a9ee2ff P d0c0d1154175432ea3332f347685aa43 [term 4 LEARNER]: Advancing to term 5
I20240501 14:10:33.034235 20695 raft_consensus.cc:979] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192: : Instructing follower 94100faea1434b27a8c6d129d1b562bc to start an election
I20240501 14:10:33.034372 20695 raft_consensus.cc:1067] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [term 2 LEADER]: Signalling peer 94100faea1434b27a8c6d129d1b562bc to start an election
I20240501 14:10:33.034871 19974 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ffd3ce2e5a1a4a0182e578d471a022ae"
dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
from {username='slave'} at 127.0.0.1:52794
I20240501 14:10:33.035032 19974 raft_consensus.cc:482] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:33.035104 19974 raft_consensus.cc:3037] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:10:33.035981 19974 raft_consensus.cc:504] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } }
I20240501 14:10:33.036363 19974 leader_election.cc:288] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [CANDIDATE]: Term 3 election: Requested vote from peers 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811), 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198:46107)
I20240501 14:10:33.036684 19821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd3ce2e5a1a4a0182e578d471a022ae" candidate_uuid: "94100faea1434b27a8c6d129d1b562bc" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "7d4b038952284c81a07423e42a349192"
I20240501 14:10:33.036698 20379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd3ce2e5a1a4a0182e578d471a022ae" candidate_uuid: "94100faea1434b27a8c6d129d1b562bc" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "5ce107fc401f4105be9947cf24c8c55e"
I20240501 14:10:33.036831 19821 raft_consensus.cc:3032] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:10:33.036831 20379 raft_consensus.cc:3037] T ffd3ce2e5a1a4a0182e578d471a022ae P 5ce107fc401f4105be9947cf24c8c55e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:10:33.036907 19821 raft_consensus.cc:729] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7d4b038952284c81a07423e42a349192, State: Running, Role: LEADER
I20240501 14:10:33.037060 19821 consensus_queue.cc:260] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } }
I20240501 14:10:33.037235 19821 raft_consensus.cc:3037] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:10:33.037966 20379 raft_consensus.cc:2454] T ffd3ce2e5a1a4a0182e578d471a022ae P 5ce107fc401f4105be9947cf24c8c55e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94100faea1434b27a8c6d129d1b562bc in term 3.
I20240501 14:10:33.038177 19821 raft_consensus.cc:2454] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94100faea1434b27a8c6d129d1b562bc in term 3.
I20240501 14:10:33.038276 19939 leader_election.cc:302] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce107fc401f4105be9947cf24c8c55e, 94100faea1434b27a8c6d129d1b562bc; no voters:
I20240501 14:10:33.038431 20690 raft_consensus.cc:2781] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:10:33.038610 20690 raft_consensus.cc:686] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [term 3 LEADER]: Becoming Leader. State: Replica: 94100faea1434b27a8c6d129d1b562bc, State: Running, Role: LEADER
I20240501 14:10:33.038744 20690 consensus_queue.cc:237] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } }
I20240501 14:10:33.040186 19714 catalog_manager.cc:5484] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc reported cstate change: term changed from 2 to 3, leader changed from 7d4b038952284c81a07423e42a349192 (127.16.13.193) to 94100faea1434b27a8c6d129d1b562bc (127.16.13.195). New cstate: current_term: 3 leader_uuid: "94100faea1434b27a8c6d129d1b562bc" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:33.120774 20551 raft_consensus.cc:979] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1: : Instructing follower 5ce107fc401f4105be9947cf24c8c55e to start an election
I20240501 14:10:33.120880 20551 raft_consensus.cc:1067] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 3 LEADER]: Signalling peer 5ce107fc401f4105be9947cf24c8c55e to start an election
I20240501 14:10:33.121378 20379 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "515893894ce0478caa58378e6c2e047b"
dest_uuid: "5ce107fc401f4105be9947cf24c8c55e"
from {username='slave'} at 127.0.0.1:49838
I20240501 14:10:33.121522 20379 raft_consensus.cc:482] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:33.121592 20379 raft_consensus.cc:3037] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:10:33.122558 20379 raft_consensus.cc:504] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } }
I20240501 14:10:33.122960 20379 leader_election.cc:288] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e [CANDIDATE]: Term 4 election: Requested vote from peers c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591), 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811)
I20240501 14:10:33.123258 19821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515893894ce0478caa58378e6c2e047b" candidate_uuid: "5ce107fc401f4105be9947cf24c8c55e" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: true dest_uuid: "7d4b038952284c81a07423e42a349192"
I20240501 14:10:33.123258 19896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515893894ce0478caa58378e6c2e047b" candidate_uuid: "5ce107fc401f4105be9947cf24c8c55e" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: true dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
I20240501 14:10:33.123438 19821 raft_consensus.cc:3037] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:10:33.123457 19896 raft_consensus.cc:3032] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:10:33.123497 19896 raft_consensus.cc:729] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c4d2a3d7c38d4794a1ee78eaef2940d1, State: Running, Role: LEADER
I20240501 14:10:33.123648 19896 consensus_queue.cc:260] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } }
I20240501 14:10:33.123812 19896 raft_consensus.cc:3037] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:10:33.124516 19821 raft_consensus.cc:2454] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce107fc401f4105be9947cf24c8c55e in term 4.
I20240501 14:10:33.124781 19896 raft_consensus.cc:2454] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce107fc401f4105be9947cf24c8c55e in term 4.
I20240501 14:10:33.124802 20345 leader_election.cc:302] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce107fc401f4105be9947cf24c8c55e, 7d4b038952284c81a07423e42a349192; no voters:
I20240501 14:10:33.124984 20626 raft_consensus.cc:2781] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:10:33.125145 20626 raft_consensus.cc:686] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e [term 4 LEADER]: Becoming Leader. State: Replica: 5ce107fc401f4105be9947cf24c8c55e, State: Running, Role: LEADER
I20240501 14:10:33.125294 20626 consensus_queue.cc:237] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } }
I20240501 14:10:33.126705 19714 catalog_manager.cc:5484] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e reported cstate change: term changed from 3 to 4, leader changed from c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194) to 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198). New cstate: current_term: 4 leader_uuid: "5ce107fc401f4105be9947cf24c8c55e" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:10:33.173991 20716 raft_consensus.cc:979] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1: : Instructing follower 7d4b038952284c81a07423e42a349192 to start an election
I20240501 14:10:33.174135 20716 raft_consensus.cc:1067] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 5 LEADER]: Signalling peer 7d4b038952284c81a07423e42a349192 to start an election
I20240501 14:10:33.174688 19821 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f796e2ca050e4666b9eb2dfd8a9ee2ff"
dest_uuid: "7d4b038952284c81a07423e42a349192"
from {username='slave'} at 127.0.0.1:51470
I20240501 14:10:33.174811 19821 raft_consensus.cc:482] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:33.174870 19821 raft_consensus.cc:3037] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:10:33.175966 19821 raft_consensus.cc:504] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } }
I20240501 14:10:33.176406 19821 leader_election.cc:288] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [CANDIDATE]: Term 6 election: Requested vote from peers c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591), 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:33.176733 19974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f796e2ca050e4666b9eb2dfd8a9ee2ff" candidate_uuid: "7d4b038952284c81a07423e42a349192" candidate_term: 6 candidate_status { last_received { term: 5 index: 29 } } ignore_live_leader: true dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
I20240501 14:10:33.176733 19896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f796e2ca050e4666b9eb2dfd8a9ee2ff" candidate_uuid: "7d4b038952284c81a07423e42a349192" candidate_term: 6 candidate_status { last_received { term: 5 index: 29 } } ignore_live_leader: true dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
I20240501 14:10:33.176884 19896 raft_consensus.cc:3032] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:10:33.176959 19896 raft_consensus.cc:729] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c4d2a3d7c38d4794a1ee78eaef2940d1, State: Running, Role: LEADER
I20240501 14:10:33.176884 19974 raft_consensus.cc:3037] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:10:33.177139 19896 consensus_queue.cc:260] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 5.29, Last appended by leader: 29, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } }
I20240501 14:10:33.177330 19896 raft_consensus.cc:3037] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:10:33.177937 19974 raft_consensus.cc:2454] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d4b038952284c81a07423e42a349192 in term 6.
I20240501 14:10:33.178048 19896 raft_consensus.cc:2454] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d4b038952284c81a07423e42a349192 in term 6.
I20240501 14:10:33.178257 19790 leader_election.cc:302] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4b038952284c81a07423e42a349192, 94100faea1434b27a8c6d129d1b562bc; no voters:
I20240501 14:10:33.178501 20695 raft_consensus.cc:2781] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 14:10:33.178575 20695 raft_consensus.cc:686] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 6 LEADER]: Becoming Leader. State: Replica: 7d4b038952284c81a07423e42a349192, State: Running, Role: LEADER
I20240501 14:10:33.178735 20695 consensus_queue.cc:237] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 5.29, Last appended by leader: 29, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } }
I20240501 14:10:33.180372 19714 catalog_manager.cc:5484] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 reported cstate change: term changed from 5 to 6, leader changed from c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194) to 7d4b038952284c81a07423e42a349192 (127.16.13.193). New cstate: current_term: 6 leader_uuid: "7d4b038952284c81a07423e42a349192" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:33.469061 20379 raft_consensus.cc:1261] T ffd3ce2e5a1a4a0182e578d471a022ae P 5ce107fc401f4105be9947cf24c8c55e [term 3 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 14:10:33.469566 20690 consensus_queue.cc:1035] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:10:33.472203 19821 raft_consensus.cc:1261] T ffd3ce2e5a1a4a0182e578d471a022ae P 7d4b038952284c81a07423e42a349192 [term 3 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 14:10:33.472723 20721 consensus_queue.cc:1035] T ffd3ce2e5a1a4a0182e578d471a022ae P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:10:33.554656 19821 raft_consensus.cc:1261] T 515893894ce0478caa58378e6c2e047b P 7d4b038952284c81a07423e42a349192 [term 4 FOLLOWER]: Refusing update from remote peer 5ce107fc401f4105be9947cf24c8c55e: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240501 14:10:33.555122 20626 consensus_queue.cc:1035] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 14:10:33.557706 19896 raft_consensus.cc:1261] T 515893894ce0478caa58378e6c2e047b P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 4 FOLLOWER]: Refusing update from remote peer 5ce107fc401f4105be9947cf24c8c55e: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240501 14:10:33.558089 20725 consensus_queue.cc:1035] T 515893894ce0478caa58378e6c2e047b P 5ce107fc401f4105be9947cf24c8c55e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 14:10:33.582479 20304 raft_consensus.cc:3037] T f796e2ca050e4666b9eb2dfd8a9ee2ff P d0c0d1154175432ea3332f347685aa43 [term 5 LEARNER]: Advancing to term 6
W20240501 14:10:33.582630 19769 auto_rebalancer.cc:253] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
I20240501 14:10:33.583513 20304 raft_consensus.cc:1261] T f796e2ca050e4666b9eb2dfd8a9ee2ff P d0c0d1154175432ea3332f347685aa43 [term 6 LEARNER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 5 index: 29. Preceding OpId from leader: term: 6 index: 30. (index mismatch)
I20240501 14:10:33.583971 20705 consensus_queue.cc:1035] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
W20240501 14:10:33.584388 19769 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 57d657b87203496ca9d47d00fd98c1af, TS 94100faea1434b27a8c6d129d1b562bc -> TS 5ce107fc401f4105be9947cf24c8c55e move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } }
W20240501 14:10:33.584439 19769 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 57d657b87203496ca9d47d00fd98c1af, TS 94100faea1434b27a8c6d129d1b562bc -> TS 5ce107fc401f4105be9947cf24c8c55e move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } }
I20240501 14:10:33.585745 20705 raft_consensus.cc:1050] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192: attempting to promote NON_VOTER d0c0d1154175432ea3332f347685aa43 to VOTER
W20240501 14:10:33.585853 20705 raft_consensus.cc:1053] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192: Unable to promote non-voter d0c0d1154175432ea3332f347685aa43: Illegal state: Leader has not yet committed an operation in its own term
W20240501 14:10:33.586529 19769 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 9d0888af56f2427c9387c3682f211a58, TS c4d2a3d7c38d4794a1ee78eaef2940d1 -> TS d0c0d1154175432ea3332f347685aa43 move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } }
W20240501 14:10:33.586577 19769 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 9d0888af56f2427c9387c3682f211a58, TS c4d2a3d7c38d4794a1ee78eaef2940d1 -> TS d0c0d1154175432ea3332f347685aa43 move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } }
W20240501 14:10:33.588193 19769 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet ffd3ce2e5a1a4a0182e578d471a022ae, TS 7d4b038952284c81a07423e42a349192 -> TS fe34f459988040e291a748133a04015c move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } }
W20240501 14:10:33.588239 19769 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet ffd3ce2e5a1a4a0182e578d471a022ae, TS 7d4b038952284c81a07423e42a349192 -> TS fe34f459988040e291a748133a04015c move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } }
W20240501 14:10:33.593483 19769 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 352498cc20094372a957c030cebf375e, TS 7d4b038952284c81a07423e42a349192 -> TS fe34f459988040e291a748133a04015c move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } }
W20240501 14:10:33.593549 19769 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 352498cc20094372a957c030cebf375e, TS 7d4b038952284c81a07423e42a349192 -> TS fe34f459988040e291a748133a04015c move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } }
W20240501 14:10:33.595075 19769 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet b8922efb08b3491fa8d947793206bf7a, TS 94100faea1434b27a8c6d129d1b562bc -> TS d0c0d1154175432ea3332f347685aa43 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
W20240501 14:10:33.595129 19769 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet b8922efb08b3491fa8d947793206bf7a, TS 94100faea1434b27a8c6d129d1b562bc -> TS d0c0d1154175432ea3332f347685aa43 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } }
I20240501 14:10:33.715806 19896 raft_consensus.cc:1261] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 6 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 5 index: 29. Preceding OpId from leader: term: 6 index: 30. (index mismatch)
I20240501 14:10:33.716269 20705 consensus_queue.cc:1035] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:10:33.719084 19974 raft_consensus.cc:1261] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 6 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 5 index: 29. Preceding OpId from leader: term: 6 index: 30. (index mismatch)
I20240501 14:10:33.719466 20731 consensus_queue.cc:1035] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:10:33.720238 20705 raft_consensus.cc:1050] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192: attempting to promote NON_VOTER d0c0d1154175432ea3332f347685aa43 to VOTER
I20240501 14:10:33.720568 20705 consensus_queue.cc:237] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 6.30, Last appended by leader: 29, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } }
I20240501 14:10:33.722239 19968 raft_consensus.cc:1261] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 6 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 6 index: 30. Preceding OpId from leader: term: 6 index: 31. (index mismatch)
I20240501 14:10:33.722280 19896 raft_consensus.cc:1261] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 6 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 6 index: 30. Preceding OpId from leader: term: 6 index: 31. (index mismatch)
I20240501 14:10:33.722662 20304 raft_consensus.cc:1261] T f796e2ca050e4666b9eb2dfd8a9ee2ff P d0c0d1154175432ea3332f347685aa43 [term 6 LEARNER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 6 index: 30. Preceding OpId from leader: term: 6 index: 31. (index mismatch)
I20240501 14:10:33.722949 20731 consensus_queue.cc:1035] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:10:33.723134 20695 consensus_queue.cc:1035] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:10:33.723431 20695 consensus_queue.cc:1035] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:10:33.725466 20695 raft_consensus.cc:2932] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 6 LEADER]: Committing config change with OpId 6.31: config changed from index 26 to 31, d0c0d1154175432ea3332f347685aa43 (127.16.13.197) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:33.725912 19968 raft_consensus.cc:2932] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 6 FOLLOWER]: Committing config change with OpId 6.31: config changed from index 26 to 31, d0c0d1154175432ea3332f347685aa43 (127.16.13.197) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:33.726996 20304 raft_consensus.cc:2932] T f796e2ca050e4666b9eb2dfd8a9ee2ff P d0c0d1154175432ea3332f347685aa43 [term 6 FOLLOWER]: Committing config change with OpId 6.31: config changed from index 26 to 31, d0c0d1154175432ea3332f347685aa43 (127.16.13.197) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:33.727946 19896 raft_consensus.cc:2932] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 6 FOLLOWER]: Committing config change with OpId 6.31: config changed from index 26 to 31, d0c0d1154175432ea3332f347685aa43 (127.16.13.197) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:33.728235 19720 catalog_manager.cc:5484] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 reported cstate change: config changed from index 26 to 31, d0c0d1154175432ea3332f347685aa43 (127.16.13.197) changed from NON_VOTER to VOTER. New cstate: current_term: 6 leader_uuid: "7d4b038952284c81a07423e42a349192" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:10:33.731961 19821 consensus_queue.cc:237] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 6.31, Last appended by leader: 29, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } }
I20240501 14:10:33.732952 20304 raft_consensus.cc:1261] T f796e2ca050e4666b9eb2dfd8a9ee2ff P d0c0d1154175432ea3332f347685aa43 [term 6 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 6 index: 31. Preceding OpId from leader: term: 6 index: 32. (index mismatch)
I20240501 14:10:33.732998 19896 raft_consensus.cc:1261] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 6 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 6 index: 31. Preceding OpId from leader: term: 6 index: 32. (index mismatch)
I20240501 14:10:33.733279 20731 consensus_queue.cc:1035] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 14:10:33.733497 20695 consensus_queue.cc:1035] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 14:10:33.734782 20695 raft_consensus.cc:2932] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 6 LEADER]: Committing config change with OpId 6.32: config changed from index 31 to 32, VOTER 94100faea1434b27a8c6d129d1b562bc (127.16.13.195) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:33.735378 19896 raft_consensus.cc:2932] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 6 FOLLOWER]: Committing config change with OpId 6.32: config changed from index 31 to 32, VOTER 94100faea1434b27a8c6d129d1b562bc (127.16.13.195) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:33.736272 20304 raft_consensus.cc:2932] T f796e2ca050e4666b9eb2dfd8a9ee2ff P d0c0d1154175432ea3332f347685aa43 [term 6 FOLLOWER]: Committing config change with OpId 6.32: config changed from index 31 to 32, VOTER 94100faea1434b27a8c6d129d1b562bc (127.16.13.195) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } }
I20240501 14:10:33.736450 19708 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f796e2ca050e4666b9eb2dfd8a9ee2ff with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:10:33.737428 19712 catalog_manager.cc:5484] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 reported cstate change: config changed from index 31 to 32, VOTER 94100faea1434b27a8c6d129d1b562bc (127.16.13.195) evicted. New cstate: current_term: 6 leader_uuid: "7d4b038952284c81a07423e42a349192" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:10:33.739660 19964 tablet_service.cc:1514] Processing DeleteTablet for tablet f796e2ca050e4666b9eb2dfd8a9ee2ff with delete_type TABLET_DATA_TOMBSTONED (TS 94100faea1434b27a8c6d129d1b562bc not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:52786
I20240501 14:10:33.740041 20738 tablet_replica.cc:330] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc: stopping tablet replica
I20240501 14:10:33.740173 20738 raft_consensus.cc:2229] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 6 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:33.740347 20738 raft_consensus.cc:2258] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc [term 6 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:33.740852 20738 ts_tablet_manager.cc:1875] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:33.742653 20738 ts_tablet_manager.cc:1888] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 6.31
I20240501 14:10:33.742717 20738 log.cc:1196] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 94100faea1434b27a8c6d129d1b562bc: Deleting WAL directory at /tmp/dist-test-task_Bxli_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714572529860089-16439-0/minicluster-data/ts-2-root/wals/f796e2ca050e4666b9eb2dfd8a9ee2ff
I20240501 14:10:33.743227 19708 catalog_manager.cc:4830] TS 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685): tablet f796e2ca050e4666b9eb2dfd8a9ee2ff (table test-workload [id=3c75157e4d6d41ec8441e28e2e87f07e]) successfully deleted
I20240501 14:10:34.741245 19968 consensus_queue.cc:237] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } }
I20240501 14:10:34.742733 19821 raft_consensus.cc:1261] T 603909ab31c7462c9e2c893184a5fc6c P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 14:10:34.742735 20227 raft_consensus.cc:1261] T 603909ab31c7462c9e2c893184a5fc6c P fe34f459988040e291a748133a04015c [term 1 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
W20240501 14:10:34.742774 19938 consensus_peers.cc:485] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc -> Peer d0c0d1154175432ea3332f347685aa43 (127.16.13.197:40045): Couldn't send request to peer d0c0d1154175432ea3332f347685aa43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 603909ab31c7462c9e2c893184a5fc6c. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:34.743130 20744 consensus_queue.cc:1035] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:10:34.743337 20743 consensus_queue.cc:1035] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:10:34.745031 20721 raft_consensus.cc:2932] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, NON_VOTER d0c0d1154175432ea3332f347685aa43 (127.16.13.197) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } } }
I20240501 14:10:34.745445 20227 raft_consensus.cc:2932] T 603909ab31c7462c9e2c893184a5fc6c P fe34f459988040e291a748133a04015c [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, NON_VOTER d0c0d1154175432ea3332f347685aa43 (127.16.13.197) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } } }
I20240501 14:10:34.745445 19821 raft_consensus.cc:2932] T 603909ab31c7462c9e2c893184a5fc6c P 7d4b038952284c81a07423e42a349192 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, NON_VOTER d0c0d1154175432ea3332f347685aa43 (127.16.13.197) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } } }
I20240501 14:10:34.747431 19823 consensus_queue.cc:237] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } }
I20240501 14:10:34.747609 19718 catalog_manager.cc:5484] T 603909ab31c7462c9e2c893184a5fc6c P fe34f459988040e291a748133a04015c reported cstate change: config changed from index 28 to 29, NON_VOTER d0c0d1154175432ea3332f347685aa43 (127.16.13.197) added. New cstate: current_term: 1 leader_uuid: "94100faea1434b27a8c6d129d1b562bc" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } } }
I20240501 14:10:34.749426 19896 raft_consensus.cc:1261] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 3 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240501 14:10:34.749745 19968 raft_consensus.cc:1261] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [term 3 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
W20240501 14:10:34.749832 19790 consensus_peers.cc:485] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 -> Peer fe34f459988040e291a748133a04015c (127.16.13.196:35779): Couldn't send request to peer fe34f459988040e291a748133a04015c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8922efb08b3491fa8d947793206bf7a. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:34.750026 20750 consensus_queue.cc:1035] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:10:34.751137 20750 consensus_queue.cc:1035] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:10:34.751852 20749 raft_consensus.cc:2932] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [term 3 LEADER]: Committing config change with OpId 3.29: config changed from index -1 to 29, NON_VOTER fe34f459988040e291a748133a04015c (127.16.13.196) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } } }
I20240501 14:10:34.752503 19896 raft_consensus.cc:2932] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index -1 to 29, NON_VOTER fe34f459988040e291a748133a04015c (127.16.13.196) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } } }
I20240501 14:10:34.754355 19968 raft_consensus.cc:2932] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index -1 to 29, NON_VOTER fe34f459988040e291a748133a04015c (127.16.13.196) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } } }
I20240501 14:10:34.754595 19712 catalog_manager.cc:5484] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 reported cstate change: config changed from index -1 to 29, NON_VOTER fe34f459988040e291a748133a04015c (127.16.13.196) added. New cstate: current_term: 3 leader_uuid: "7d4b038952284c81a07423e42a349192" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } } }
I20240501 14:10:34.754989 19896 consensus_queue.cc:237] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } }
W20240501 14:10:34.756594 19864 consensus_peers.cc:485] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 -> Peer 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198:46107): Couldn't send request to peer 5ce107fc401f4105be9947cf24c8c55e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57d657b87203496ca9d47d00fd98c1af. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:34.756884 20227 raft_consensus.cc:1261] T 57d657b87203496ca9d47d00fd98c1af P fe34f459988040e291a748133a04015c [term 1 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 14:10:34.756984 19968 raft_consensus.cc:1261] T 57d657b87203496ca9d47d00fd98c1af P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 14:10:34.757331 20758 consensus_queue.cc:1035] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:10:34.757567 20716 consensus_queue.cc:1035] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:10:34.759079 20759 raft_consensus.cc:2932] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } } }
I20240501 14:10:34.759490 19968 raft_consensus.cc:2932] T 57d657b87203496ca9d47d00fd98c1af P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } } }
I20240501 14:10:34.759479 20227 raft_consensus.cc:2932] T 57d657b87203496ca9d47d00fd98c1af P fe34f459988040e291a748133a04015c [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } } }
I20240501 14:10:34.761520 19714 catalog_manager.cc:5484] T 57d657b87203496ca9d47d00fd98c1af P 94100faea1434b27a8c6d129d1b562bc reported cstate change: config changed from index 29 to 30, NON_VOTER 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) added. New cstate: current_term: 1 leader_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } } }
I20240501 14:10:34.762161 19821 consensus_queue.cc:237] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 6.32, Last appended by leader: 29, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } }
I20240501 14:10:34.763623 20304 raft_consensus.cc:1261] T f796e2ca050e4666b9eb2dfd8a9ee2ff P d0c0d1154175432ea3332f347685aa43 [term 6 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 6 index: 32. Preceding OpId from leader: term: 6 index: 33. (index mismatch)
W20240501 14:10:34.764168 19788 consensus_peers.cc:485] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 -> Peer 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198:46107): Couldn't send request to peer 5ce107fc401f4105be9947cf24c8c55e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f796e2ca050e4666b9eb2dfd8a9ee2ff. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:34.764398 20749 consensus_queue.cc:1035] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 14:10:34.764475 19896 raft_consensus.cc:1261] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 6 FOLLOWER]: Refusing update from remote peer 7d4b038952284c81a07423e42a349192: Log matching property violated. Preceding OpId in replica: term: 6 index: 32. Preceding OpId from leader: term: 6 index: 33. (index mismatch)
I20240501 14:10:34.764736 20749 consensus_queue.cc:1035] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 14:10:34.766134 20751 raft_consensus.cc:2932] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 [term 6 LEADER]: Committing config change with OpId 6.33: config changed from index 32 to 33, NON_VOTER 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } } }
I20240501 14:10:34.766628 20304 raft_consensus.cc:2932] T f796e2ca050e4666b9eb2dfd8a9ee2ff P d0c0d1154175432ea3332f347685aa43 [term 6 FOLLOWER]: Committing config change with OpId 6.33: config changed from index 32 to 33, NON_VOTER 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } } }
I20240501 14:10:34.768543 20379 consensus_queue.cc:237] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } }
I20240501 14:10:34.768568 19718 catalog_manager.cc:5484] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192 reported cstate change: config changed from index 32 to 33, NON_VOTER 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) added. New cstate: current_term: 6 leader_uuid: "7d4b038952284c81a07423e42a349192" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:34.769836 19896 raft_consensus.cc:2932] T f796e2ca050e4666b9eb2dfd8a9ee2ff P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 6 FOLLOWER]: Committing config change with OpId 6.33: config changed from index 32 to 33, NON_VOTER 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: NON_VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: true } } }
I20240501 14:10:34.770423 19821 raft_consensus.cc:1261] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Refusing update from remote peer 5ce107fc401f4105be9947cf24c8c55e: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 14:10:34.770766 20740 consensus_queue.cc:1035] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:10:34.772783 19968 raft_consensus.cc:1261] T 352498cc20094372a957c030cebf375e P 94100faea1434b27a8c6d129d1b562bc [term 2 FOLLOWER]: Refusing update from remote peer 5ce107fc401f4105be9947cf24c8c55e: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 14:10:34.773243 20740 consensus_queue.cc:1035] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:10:34.774516 20740 raft_consensus.cc:2932] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER d0c0d1154175432ea3332f347685aa43 (127.16.13.197) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } } }
I20240501 14:10:34.774999 19821 raft_consensus.cc:2932] T 352498cc20094372a957c030cebf375e P 7d4b038952284c81a07423e42a349192 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER d0c0d1154175432ea3332f347685aa43 (127.16.13.197) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } } }
W20240501 14:10:34.776309 20342 consensus_peers.cc:485] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e -> Peer d0c0d1154175432ea3332f347685aa43 (127.16.13.197:40045): Couldn't send request to peer d0c0d1154175432ea3332f347685aa43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 352498cc20094372a957c030cebf375e. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:34.776541 19968 raft_consensus.cc:2932] T 352498cc20094372a957c030cebf375e P 94100faea1434b27a8c6d129d1b562bc [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER d0c0d1154175432ea3332f347685aa43 (127.16.13.197) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } } }
I20240501 14:10:34.777662 19896 consensus_queue.cc:237] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } }
I20240501 14:10:34.778517 19718 catalog_manager.cc:5484] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e reported cstate change: config changed from index 29 to 31, NON_VOTER d0c0d1154175432ea3332f347685aa43 (127.16.13.197) added. New cstate: current_term: 2 leader_uuid: "5ce107fc401f4105be9947cf24c8c55e" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ce107fc401f4105be9947cf24c8c55e" member_type: VOTER last_known_addr { host: "127.16.13.198" port: 46107 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: NON_VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:34.778995 20304 raft_consensus.cc:1261] T 4b28da769e1f46e1b05bd25e8428316f P d0c0d1154175432ea3332f347685aa43 [term 1 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
W20240501 14:10:34.779363 19866 consensus_peers.cc:485] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 -> Peer fe34f459988040e291a748133a04015c (127.16.13.196:35779): Couldn't send request to peer fe34f459988040e291a748133a04015c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b28da769e1f46e1b05bd25e8428316f. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:34.779407 20716 consensus_queue.cc:1035] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:10:34.780262 19968 raft_consensus.cc:1261] T 4b28da769e1f46e1b05bd25e8428316f P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 14:10:34.780588 20716 consensus_queue.cc:1035] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:10:34.781107 20716 raft_consensus.cc:2932] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER fe34f459988040e291a748133a04015c (127.16.13.196) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } } }
I20240501 14:10:34.783015 20304 raft_consensus.cc:2932] T 4b28da769e1f46e1b05bd25e8428316f P d0c0d1154175432ea3332f347685aa43 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER fe34f459988040e291a748133a04015c (127.16.13.196) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } } }
I20240501 14:10:34.783213 19714 catalog_manager.cc:5484] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1 reported cstate change: config changed from index 29 to 30, NON_VOTER fe34f459988040e291a748133a04015c (127.16.13.196) added. New cstate: current_term: 1 leader_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:34.785403 19968 raft_consensus.cc:2932] T 4b28da769e1f46e1b05bd25e8428316f P 94100faea1434b27a8c6d129d1b562bc [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER fe34f459988040e291a748133a04015c (127.16.13.196) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } attrs { replace: true } } peers { permanent_uuid: "d0c0d1154175432ea3332f347685aa43" member_type: VOTER last_known_addr { host: "127.16.13.197" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } } }
I20240501 14:10:34.831555 19775 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:10:34.833077 19968 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9d0888af56f2427c9387c3682f211a58"
dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
mode: GRACEFUL
new_leader_uuid: "fe34f459988040e291a748133a04015c"
from {username='slave'} at 127.0.0.1:52850
I20240501 14:10:34.833240 19968 raft_consensus.cc:595] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [term 2 LEADER]: Received request to transfer leadership to TS fe34f459988040e291a748133a04015c
I20240501 14:10:34.833950 19821 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b8922efb08b3491fa8d947793206bf7a"
dest_uuid: "7d4b038952284c81a07423e42a349192"
mode: GRACEFUL
new_leader_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
from {username='slave'} at 127.0.0.1:51516
I20240501 14:10:34.834071 19821 raft_consensus.cc:595] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [term 3 LEADER]: Received request to transfer leadership to TS c4d2a3d7c38d4794a1ee78eaef2940d1
I20240501 14:10:34.834381 19775 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240501 14:10:34.834509 19775 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:10:34.998093 20744 raft_consensus.cc:979] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc: : Instructing follower fe34f459988040e291a748133a04015c to start an election
I20240501 14:10:34.998226 20744 raft_consensus.cc:1067] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [term 2 LEADER]: Signalling peer fe34f459988040e291a748133a04015c to start an election
I20240501 14:10:34.998598 20227 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9d0888af56f2427c9387c3682f211a58"
dest_uuid: "fe34f459988040e291a748133a04015c"
from {username='slave'} at 127.0.0.1:47178
I20240501 14:10:34.998721 20227 raft_consensus.cc:482] T 9d0888af56f2427c9387c3682f211a58 P fe34f459988040e291a748133a04015c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:34.998778 20227 raft_consensus.cc:3037] T 9d0888af56f2427c9387c3682f211a58 P fe34f459988040e291a748133a04015c [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:10:34.999720 20227 raft_consensus.cc:504] T 9d0888af56f2427c9387c3682f211a58 P fe34f459988040e291a748133a04015c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } }
I20240501 14:10:35.000187 20227 leader_election.cc:288] T 9d0888af56f2427c9387c3682f211a58 P fe34f459988040e291a748133a04015c [CANDIDATE]: Term 3 election: Requested vote from peers c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591), 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:35.000453 19896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0888af56f2427c9387c3682f211a58" candidate_uuid: "fe34f459988040e291a748133a04015c" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
I20240501 14:10:35.000573 19896 raft_consensus.cc:3037] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:10:35.000633 19968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0888af56f2427c9387c3682f211a58" candidate_uuid: "fe34f459988040e291a748133a04015c" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
I20240501 14:10:35.000728 19968 raft_consensus.cc:3032] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:10:35.000771 19968 raft_consensus.cc:729] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 94100faea1434b27a8c6d129d1b562bc, State: Running, Role: LEADER
I20240501 14:10:35.000895 19968 consensus_queue.cc:260] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } }
I20240501 14:10:35.001044 19968 raft_consensus.cc:3037] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:10:35.001628 19896 raft_consensus.cc:2454] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe34f459988040e291a748133a04015c in term 3.
I20240501 14:10:35.001986 19968 raft_consensus.cc:2454] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe34f459988040e291a748133a04015c in term 3.
I20240501 14:10:35.002292 20194 leader_election.cc:302] T 9d0888af56f2427c9387c3682f211a58 P fe34f459988040e291a748133a04015c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94100faea1434b27a8c6d129d1b562bc, fe34f459988040e291a748133a04015c; no voters:
I20240501 14:10:35.002507 20701 raft_consensus.cc:2781] T 9d0888af56f2427c9387c3682f211a58 P fe34f459988040e291a748133a04015c [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:10:35.002745 20701 raft_consensus.cc:686] T 9d0888af56f2427c9387c3682f211a58 P fe34f459988040e291a748133a04015c [term 3 LEADER]: Becoming Leader. State: Replica: fe34f459988040e291a748133a04015c, State: Running, Role: LEADER
I20240501 14:10:35.002903 20701 consensus_queue.cc:237] T 9d0888af56f2427c9387c3682f211a58 P fe34f459988040e291a748133a04015c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } }
I20240501 14:10:35.004388 19712 catalog_manager.cc:5484] T 9d0888af56f2427c9387c3682f211a58 P fe34f459988040e291a748133a04015c reported cstate change: term changed from 2 to 3, leader changed from 94100faea1434b27a8c6d129d1b562bc (127.16.13.195) to fe34f459988040e291a748133a04015c (127.16.13.196). New cstate: current_term: 3 leader_uuid: "fe34f459988040e291a748133a04015c" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:10:35.132133 20777 ts_tablet_manager.cc:902] T 57d657b87203496ca9d47d00fd98c1af P 5ce107fc401f4105be9947cf24c8c55e: Initiating tablet copy from peer c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591)
I20240501 14:10:35.132830 20777 tablet_copy_client.cc:308] T 57d657b87203496ca9d47d00fd98c1af P 5ce107fc401f4105be9947cf24c8c55e: tablet copy: Beginning tablet copy session from remote peer at address 127.16.13.194:39591
I20240501 14:10:35.133342 19910 tablet_copy_service.cc:140] P c4d2a3d7c38d4794a1ee78eaef2940d1: Received BeginTabletCopySession request for tablet 57d657b87203496ca9d47d00fd98c1af from peer 5ce107fc401f4105be9947cf24c8c55e ({username='slave'} at 127.0.0.1:56836)
I20240501 14:10:35.133450 19910 tablet_copy_service.cc:161] P c4d2a3d7c38d4794a1ee78eaef2940d1: Beginning new tablet copy session on tablet 57d657b87203496ca9d47d00fd98c1af from peer 5ce107fc401f4105be9947cf24c8c55e at {username='slave'} at 127.0.0.1:56836: session id = 5ce107fc401f4105be9947cf24c8c55e-57d657b87203496ca9d47d00fd98c1af
I20240501 14:10:35.134479 19910 tablet_copy_source_session.cc:205] T 57d657b87203496ca9d47d00fd98c1af P c4d2a3d7c38d4794a1ee78eaef2940d1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:35.135341 20777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57d657b87203496ca9d47d00fd98c1af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:35.137881 20777 tablet_copy_client.cc:783] T 57d657b87203496ca9d47d00fd98c1af P 5ce107fc401f4105be9947cf24c8c55e: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:35.138083 20777 tablet_copy_client.cc:647] T 57d657b87203496ca9d47d00fd98c1af P 5ce107fc401f4105be9947cf24c8c55e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:35.139349 20781 tablet_copy_client.cc:1002] T 57d657b87203496ca9d47d00fd98c1af P 5ce107fc401f4105be9947cf24c8c55e: tablet copy: Injecting latency into file download: 2000
I20240501 14:10:35.157685 20782 ts_tablet_manager.cc:902] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 5ce107fc401f4105be9947cf24c8c55e: Initiating tablet copy from peer 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811)
I20240501 14:10:35.158358 20782 tablet_copy_client.cc:308] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 5ce107fc401f4105be9947cf24c8c55e: tablet copy: Beginning tablet copy session from remote peer at address 127.16.13.193:38811
I20240501 14:10:35.158797 19834 tablet_copy_service.cc:140] P 7d4b038952284c81a07423e42a349192: Received BeginTabletCopySession request for tablet f796e2ca050e4666b9eb2dfd8a9ee2ff from peer 5ce107fc401f4105be9947cf24c8c55e ({username='slave'} at 127.0.0.1:51512)
I20240501 14:10:35.158900 19834 tablet_copy_service.cc:161] P 7d4b038952284c81a07423e42a349192: Beginning new tablet copy session on tablet f796e2ca050e4666b9eb2dfd8a9ee2ff from peer 5ce107fc401f4105be9947cf24c8c55e at {username='slave'} at 127.0.0.1:51512: session id = 5ce107fc401f4105be9947cf24c8c55e-f796e2ca050e4666b9eb2dfd8a9ee2ff
I20240501 14:10:35.159848 19834 tablet_copy_source_session.cc:205] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 7d4b038952284c81a07423e42a349192: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:35.160674 20782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f796e2ca050e4666b9eb2dfd8a9ee2ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:35.162637 20782 tablet_copy_client.cc:783] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 5ce107fc401f4105be9947cf24c8c55e: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:35.162823 20782 tablet_copy_client.cc:647] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 5ce107fc401f4105be9947cf24c8c55e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:35.163817 20786 tablet_copy_client.cc:1002] T f796e2ca050e4666b9eb2dfd8a9ee2ff P 5ce107fc401f4105be9947cf24c8c55e: tablet copy: Injecting latency into file download: 2000
I20240501 14:10:35.232501 20750 raft_consensus.cc:979] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192: : Instructing follower c4d2a3d7c38d4794a1ee78eaef2940d1 to start an election
I20240501 14:10:35.232638 20749 raft_consensus.cc:1067] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [term 3 LEADER]: Signalling peer c4d2a3d7c38d4794a1ee78eaef2940d1 to start an election
I20240501 14:10:35.233031 19896 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8922efb08b3491fa8d947793206bf7a"
dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
from {username='slave'} at 127.0.0.1:56780
I20240501 14:10:35.233157 19896 raft_consensus.cc:482] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:35.233222 19896 raft_consensus.cc:3037] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:10:35.234097 19896 raft_consensus.cc:504] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } }
I20240501 14:10:35.234459 19896 leader_election.cc:288] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 4 election: Requested vote from peers 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811), 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:35.234727 19821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8922efb08b3491fa8d947793206bf7a" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "7d4b038952284c81a07423e42a349192"
I20240501 14:10:35.234774 19974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8922efb08b3491fa8d947793206bf7a" candidate_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
I20240501 14:10:35.234905 19974 raft_consensus.cc:3037] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:10:35.234886 19821 raft_consensus.cc:3032] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:10:35.234977 19821 raft_consensus.cc:729] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7d4b038952284c81a07423e42a349192, State: Running, Role: LEADER
I20240501 14:10:35.235113 19821 consensus_queue.cc:260] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } }
I20240501 14:10:35.235319 19821 raft_consensus.cc:3037] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:10:35.235877 19974 raft_consensus.cc:2454] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 4.
I20240501 14:10:35.236119 19866 leader_election.cc:302] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94100faea1434b27a8c6d129d1b562bc, c4d2a3d7c38d4794a1ee78eaef2940d1; no voters:
I20240501 14:10:35.236256 19821 raft_consensus.cc:2454] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d2a3d7c38d4794a1ee78eaef2940d1 in term 4.
I20240501 14:10:35.236274 20758 raft_consensus.cc:2781] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:10:35.236387 20758 raft_consensus.cc:686] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 4 LEADER]: Becoming Leader. State: Replica: c4d2a3d7c38d4794a1ee78eaef2940d1, State: Running, Role: LEADER
I20240501 14:10:35.236532 20758 consensus_queue.cc:237] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } }
I20240501 14:10:35.237962 19712 catalog_manager.cc:5484] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 reported cstate change: term changed from 3 to 4, leader changed from 7d4b038952284c81a07423e42a349192 (127.16.13.193) to c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194). New cstate: current_term: 4 leader_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:35.244132 19896 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b8922efb08b3491fa8d947793206bf7a"
dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56764
I20240501 14:10:35.244252 19896 raft_consensus.cc:595] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 4 LEADER]: Received request to transfer leadership
I20240501 14:10:35.249317 19896 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b8922efb08b3491fa8d947793206bf7a"
dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56764
I20240501 14:10:35.249439 19896 raft_consensus.cc:595] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 4 LEADER]: Received request to transfer leadership
W20240501 14:10:35.249658 19706 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for b8922efb08b3491fa8d947793206bf7a already in progress
I20240501 14:10:35.250305 19896 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b8922efb08b3491fa8d947793206bf7a"
dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56764
I20240501 14:10:35.250406 19896 raft_consensus.cc:595] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 4 LEADER]: Received request to transfer leadership
W20240501 14:10:35.250707 19769 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for b8922efb08b3491fa8d947793206bf7a already in progress
W20240501 14:10:35.250766 19769 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for b8922efb08b3491fa8d947793206bf7a already in progress
I20240501 14:10:35.256844 20788 ts_tablet_manager.cc:902] T 4b28da769e1f46e1b05bd25e8428316f P fe34f459988040e291a748133a04015c: Initiating tablet copy from peer c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591)
I20240501 14:10:35.257440 20788 tablet_copy_client.cc:308] T 4b28da769e1f46e1b05bd25e8428316f P fe34f459988040e291a748133a04015c: tablet copy: Beginning tablet copy session from remote peer at address 127.16.13.194:39591
I20240501 14:10:35.257823 19910 tablet_copy_service.cc:140] P c4d2a3d7c38d4794a1ee78eaef2940d1: Received BeginTabletCopySession request for tablet 4b28da769e1f46e1b05bd25e8428316f from peer fe34f459988040e291a748133a04015c ({username='slave'} at 127.0.0.1:56854)
I20240501 14:10:35.257906 19910 tablet_copy_service.cc:161] P c4d2a3d7c38d4794a1ee78eaef2940d1: Beginning new tablet copy session on tablet 4b28da769e1f46e1b05bd25e8428316f from peer fe34f459988040e291a748133a04015c at {username='slave'} at 127.0.0.1:56854: session id = fe34f459988040e291a748133a04015c-4b28da769e1f46e1b05bd25e8428316f
I20240501 14:10:35.258862 19910 tablet_copy_source_session.cc:205] T 4b28da769e1f46e1b05bd25e8428316f P c4d2a3d7c38d4794a1ee78eaef2940d1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:35.259692 20788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b28da769e1f46e1b05bd25e8428316f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:35.261806 20788 tablet_copy_client.cc:783] T 4b28da769e1f46e1b05bd25e8428316f P fe34f459988040e291a748133a04015c: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:35.261957 20788 tablet_copy_client.cc:647] T 4b28da769e1f46e1b05bd25e8428316f P fe34f459988040e291a748133a04015c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:35.262878 20792 tablet_copy_client.cc:1002] T 4b28da769e1f46e1b05bd25e8428316f P fe34f459988040e291a748133a04015c: tablet copy: Injecting latency into file download: 2000
I20240501 14:10:35.281877 20793 ts_tablet_manager.cc:902] T 603909ab31c7462c9e2c893184a5fc6c P d0c0d1154175432ea3332f347685aa43: Initiating tablet copy from peer 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
I20240501 14:10:35.282462 20793 tablet_copy_client.cc:308] T 603909ab31c7462c9e2c893184a5fc6c P d0c0d1154175432ea3332f347685aa43: tablet copy: Beginning tablet copy session from remote peer at address 127.16.13.195:36685
I20240501 14:10:35.282896 19985 tablet_copy_service.cc:140] P 94100faea1434b27a8c6d129d1b562bc: Received BeginTabletCopySession request for tablet 603909ab31c7462c9e2c893184a5fc6c from peer d0c0d1154175432ea3332f347685aa43 ({username='slave'} at 127.0.0.1:52834)
I20240501 14:10:35.283006 19985 tablet_copy_service.cc:161] P 94100faea1434b27a8c6d129d1b562bc: Beginning new tablet copy session on tablet 603909ab31c7462c9e2c893184a5fc6c from peer d0c0d1154175432ea3332f347685aa43 at {username='slave'} at 127.0.0.1:52834: session id = d0c0d1154175432ea3332f347685aa43-603909ab31c7462c9e2c893184a5fc6c
I20240501 14:10:35.284196 19985 tablet_copy_source_session.cc:205] T 603909ab31c7462c9e2c893184a5fc6c P 94100faea1434b27a8c6d129d1b562bc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:35.285135 20793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 603909ab31c7462c9e2c893184a5fc6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:35.287211 20793 tablet_copy_client.cc:783] T 603909ab31c7462c9e2c893184a5fc6c P d0c0d1154175432ea3332f347685aa43: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:35.287362 20793 tablet_copy_client.cc:647] T 603909ab31c7462c9e2c893184a5fc6c P d0c0d1154175432ea3332f347685aa43: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:35.288551 20797 tablet_copy_client.cc:1002] T 603909ab31c7462c9e2c893184a5fc6c P d0c0d1154175432ea3332f347685aa43: tablet copy: Injecting latency into file download: 2000
I20240501 14:10:35.331246 20798 ts_tablet_manager.cc:902] T 352498cc20094372a957c030cebf375e P d0c0d1154175432ea3332f347685aa43: Initiating tablet copy from peer 5ce107fc401f4105be9947cf24c8c55e (127.16.13.198:46107)
I20240501 14:10:35.331934 20798 tablet_copy_client.cc:308] T 352498cc20094372a957c030cebf375e P d0c0d1154175432ea3332f347685aa43: tablet copy: Beginning tablet copy session from remote peer at address 127.16.13.198:46107
I20240501 14:10:35.335966 20389 tablet_copy_service.cc:140] P 5ce107fc401f4105be9947cf24c8c55e: Received BeginTabletCopySession request for tablet 352498cc20094372a957c030cebf375e from peer d0c0d1154175432ea3332f347685aa43 ({username='slave'} at 127.0.0.1:52700)
I20240501 14:10:35.336084 20389 tablet_copy_service.cc:161] P 5ce107fc401f4105be9947cf24c8c55e: Beginning new tablet copy session on tablet 352498cc20094372a957c030cebf375e from peer d0c0d1154175432ea3332f347685aa43 at {username='slave'} at 127.0.0.1:52700: session id = d0c0d1154175432ea3332f347685aa43-352498cc20094372a957c030cebf375e
I20240501 14:10:35.337054 20389 tablet_copy_source_session.cc:205] T 352498cc20094372a957c030cebf375e P 5ce107fc401f4105be9947cf24c8c55e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:35.337749 20798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 352498cc20094372a957c030cebf375e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:35.339967 20798 tablet_copy_client.cc:783] T 352498cc20094372a957c030cebf375e P d0c0d1154175432ea3332f347685aa43: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:35.340122 20798 tablet_copy_client.cc:647] T 352498cc20094372a957c030cebf375e P d0c0d1154175432ea3332f347685aa43: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:35.341162 20802 tablet_copy_client.cc:1002] T 352498cc20094372a957c030cebf375e P d0c0d1154175432ea3332f347685aa43: tablet copy: Injecting latency into file download: 2000
I20240501 14:10:35.411074 19896 raft_consensus.cc:1261] T 9d0888af56f2427c9387c3682f211a58 P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 3 FOLLOWER]: Refusing update from remote peer fe34f459988040e291a748133a04015c: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 14:10:35.411567 20701 consensus_queue.cc:1035] T 9d0888af56f2427c9387c3682f211a58 P fe34f459988040e291a748133a04015c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:10:35.414192 19974 raft_consensus.cc:1261] T 9d0888af56f2427c9387c3682f211a58 P 94100faea1434b27a8c6d129d1b562bc [term 3 FOLLOWER]: Refusing update from remote peer fe34f459988040e291a748133a04015c: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 14:10:35.414645 20775 consensus_queue.cc:1035] T 9d0888af56f2427c9387c3682f211a58 P fe34f459988040e291a748133a04015c [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:10:35.685277 19968 raft_consensus.cc:1261] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [term 4 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240501 14:10:35.685647 20810 consensus_queue.cc:1035] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:10:35.687067 20716 raft_consensus.cc:979] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1: : Instructing follower 94100faea1434b27a8c6d129d1b562bc to start an election
I20240501 14:10:35.687160 20716 raft_consensus.cc:1067] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 4 LEADER]: Signalling peer 94100faea1434b27a8c6d129d1b562bc to start an election
I20240501 14:10:35.688009 19974 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8922efb08b3491fa8d947793206bf7a"
dest_uuid: "94100faea1434b27a8c6d129d1b562bc"
from {username='slave'} at 127.0.0.1:52796
I20240501 14:10:35.688134 19974 raft_consensus.cc:482] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:35.688195 19974 raft_consensus.cc:3037] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:10:35.688412 19823 raft_consensus.cc:1261] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [term 4 FOLLOWER]: Refusing update from remote peer c4d2a3d7c38d4794a1ee78eaef2940d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240501 14:10:35.688788 20716 consensus_queue.cc:1035] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
W20240501 14:10:35.688954 19866 consensus_peers.cc:485] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 -> Peer fe34f459988040e291a748133a04015c (127.16.13.196:35779): Couldn't send request to peer fe34f459988040e291a748133a04015c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8922efb08b3491fa8d947793206bf7a. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:35.689455 19974 raft_consensus.cc:504] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } }
I20240501 14:10:35.689819 19974 leader_election.cc:288] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [CANDIDATE]: Term 5 election: Requested vote from peers 7d4b038952284c81a07423e42a349192 (127.16.13.193:38811), c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194:39591)
I20240501 14:10:35.691448 19896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8922efb08b3491fa8d947793206bf7a" candidate_uuid: "94100faea1434b27a8c6d129d1b562bc" candidate_term: 5 candidate_status { last_received { term: 4 index: 30 } } ignore_live_leader: true dest_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1"
I20240501 14:10:35.691632 19823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8922efb08b3491fa8d947793206bf7a" candidate_uuid: "94100faea1434b27a8c6d129d1b562bc" candidate_term: 5 candidate_status { last_received { term: 4 index: 30 } } ignore_live_leader: true dest_uuid: "7d4b038952284c81a07423e42a349192"
I20240501 14:10:35.691807 19896 raft_consensus.cc:3032] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:10:35.691854 19896 raft_consensus.cc:729] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c4d2a3d7c38d4794a1ee78eaef2940d1, State: Running, Role: LEADER
I20240501 14:10:35.691859 19823 raft_consensus.cc:3037] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:10:35.692003 19896 consensus_queue.cc:260] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 30, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } }
I20240501 14:10:35.692189 19896 raft_consensus.cc:3037] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:10:35.692941 19823 raft_consensus.cc:2454] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94100faea1434b27a8c6d129d1b562bc in term 5.
I20240501 14:10:35.693023 19896 raft_consensus.cc:2454] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94100faea1434b27a8c6d129d1b562bc in term 5.
I20240501 14:10:35.693203 19941 leader_election.cc:302] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4b038952284c81a07423e42a349192, 94100faea1434b27a8c6d129d1b562bc; no voters:
I20240501 14:10:35.693403 20744 raft_consensus.cc:2781] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:10:35.693573 20744 raft_consensus.cc:686] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [term 5 LEADER]: Becoming Leader. State: Replica: 94100faea1434b27a8c6d129d1b562bc, State: Running, Role: LEADER
I20240501 14:10:35.693719 20744 consensus_queue.cc:237] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 30, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } }
I20240501 14:10:35.695019 19712 catalog_manager.cc:5484] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc reported cstate change: term changed from 4 to 5, leader changed from c4d2a3d7c38d4794a1ee78eaef2940d1 (127.16.13.194) to 94100faea1434b27a8c6d129d1b562bc (127.16.13.195). New cstate: current_term: 5 leader_uuid: "94100faea1434b27a8c6d129d1b562bc" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94100faea1434b27a8c6d129d1b562bc" member_type: VOTER last_known_addr { host: "127.16.13.195" port: 36685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe34f459988040e291a748133a04015c" member_type: NON_VOTER last_known_addr { host: "127.16.13.196" port: 35779 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:36.109408 19823 raft_consensus.cc:1261] T b8922efb08b3491fa8d947793206bf7a P 7d4b038952284c81a07423e42a349192 [term 5 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
I20240501 14:10:36.109827 20813 consensus_queue.cc:1035] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b038952284c81a07423e42a349192" member_type: VOTER last_known_addr { host: "127.16.13.193" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
W20240501 14:10:36.110782 19941 consensus_peers.cc:485] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc -> Peer fe34f459988040e291a748133a04015c (127.16.13.196:35779): Couldn't send request to peer fe34f459988040e291a748133a04015c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8922efb08b3491fa8d947793206bf7a. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:36.113557 19896 raft_consensus.cc:1261] T b8922efb08b3491fa8d947793206bf7a P c4d2a3d7c38d4794a1ee78eaef2940d1 [term 5 FOLLOWER]: Refusing update from remote peer 94100faea1434b27a8c6d129d1b562bc: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
I20240501 14:10:36.113902 20812 consensus_queue.cc:1035] T b8922efb08b3491fa8d947793206bf7a P 94100faea1434b27a8c6d129d1b562bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d2a3d7c38d4794a1ee78eaef2940d1" member_type: VOTER last_known_addr { host: "127.16.13.194" port: 39591 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:10:36.611840 20817 ts_tablet_manager.cc:902] T b8922efb08b3491fa8d947793206bf7a P fe34f459988040e291a748133a04015c: Initiating tablet copy from peer 94100faea1434b27a8c6d129d1b562bc (127.16.13.195:36685)
F20240501 14:10:36.612123 20817 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-001cf18440d945b6824493c39b1d7487.queue_time_us, tablet-download-pool-001cf18440d945b6824493c39b1d7487.queue_time_us) (tablet-download-pool-001cf18440d945b6824493c39b1d7487.run_cpu_time_us, tablet-download-pool-001cf18440d945b6824493c39b1d7487.run_cpu_time_us) (tablet-download-pool-001cf18440d945b6824493c39b1d7487.run_wall_time_us, tablet-download-pool-001cf18440d945b6824493c39b1d7487.run_wall_time_us) (tablet-download-pool-2e6457de95c64fb19ae1d294c436c8f7.queue_time_us, tablet-download-pool-2e6457de95c64fb19ae1d294c436c8f7.queue_time_us) (tablet-download-pool-2e6457de95c64fb19ae1d294c436c8f7.run_cpu_time_us, tablet-download-pool-2e6457de95c64fb19ae1d294c436c8f7.run_cpu_time_us) (tablet-download-pool-2e6457de95c64fb19ae1d294c436c8f7.run_wall_time_us, tablet-download-pool-2e6457de95c64fb19ae1d294c436c8f7.run_wall_time_us) (tablet-download-pool-352498cc20094372a957c030cebf375e.queue_time_us, tablet-download-pool-352498cc20094372a957c030cebf375e.queue_time_us) (tablet-download-pool-352498cc20094372a957c030cebf375e.run_cpu_time_us, tablet-download-pool-352498cc20094372a957c030cebf375e.run_cpu_time_us) (tablet-download-pool-352498cc20094372a957c030cebf375e.run_wall_time_us, tablet-download-pool-352498cc20094372a957c030cebf375e.run_wall_time_us) (tablet-download-pool-4b28da769e1f46e1b05bd25e8428316f.queue_time_us, tablet-download-pool-4b28da769e1f46e1b05bd25e8428316f.queue_time_us) (tablet-download-pool-4b28da769e1f46e1b05bd25e8428316f.run_cpu_time_us, tablet-download-pool-4b28da769e1f46e1b05bd25e8428316f.run_cpu_time_us) (tablet-download-pool-4b28da769e1f46e1b05bd25e8428316f.run_wall_time_us, tablet-download-pool-4b28da769e1f46e1b05bd25e8428316f.run_wall_time_us) (tablet-download-pool-515893894ce0478caa58378e6c2e047b.queue_time_us, tablet-download-pool-515893894ce0478caa58378e6c2e047b.queue_time_us) (tablet-download-pool-515893894ce0478caa58378e6c2e047b.run_cpu_time_us, tablet-download-pool-515893894ce0478caa58378e6c2e047b.run_cpu_time_us) (tablet-download-pool-515893894ce0478caa58378e6c2e047b.run_wall_time_us, tablet-download-pool-515893894ce0478caa58378e6c2e047b.run_wall_time_us) (tablet-download-pool-57d657b87203496ca9d47d00fd98c1af.queue_time_us, tablet-download-pool-57d657b87203496ca9d47d00fd98c1af.queue_time_us) (tablet-download-pool-57d657b87203496ca9d47d00fd98c1af.run_cpu_time_us, tablet-download-pool-57d657b87203496ca9d47d00fd98c1af.run_cpu_time_us) (tablet-download-pool-57d657b87203496ca9d47d00fd98c1af.run_wall_time_us, tablet-download-pool-57d657b87203496ca9d47d00fd98c1af.run_wall_time_us) (tablet-download-pool-603909ab31c7462c9e2c893184a5fc6c.queue_time_us, tablet-download-pool-603909ab31c7462c9e2c893184a5fc6c.queue_time_us) (tablet-download-pool-603909ab31c7462c9e2c893184a5fc6c.run_cpu_time_us, tablet-download-pool-603909ab31c7462c9e2c893184a5fc6c.run_cpu_time_us) (tablet-download-pool-603909ab31c7462c9e2c893184a5fc6c.run_wall_time_us, tablet-download-pool-603909ab31c7462c9e2c893184a5fc6c.run_wall_time_us) (tablet-download-pool-9d0888af56f2427c9387c3682f211a58.queue_time_us, tablet-download-pool-9d0888af56f2427c9387c3682f211a58.queue_time_us) (tablet-download-pool-9d0888af56f2427c9387c3682f211a58.run_cpu_time_us, tablet-download-pool-9d0888af56f2427c9387c3682f211a58.run_cpu_time_us) (tablet-download-pool-9d0888af56f2427c9387c3682f211a58.run_wall_time_us, tablet-download-pool-9d0888af56f2427c9387c3682f211a58.run_wall_time_us) (tablet-download-pool-a47e39994ef3497e96bedd8a36133e00.queue_time_us, tablet-download-pool-a47e39994ef3497e96bedd8a36133e00.queue_time_us) (tablet-download-pool-a47e39994ef3497e96bedd8a36133e00.run_cpu_time_us, tablet-download-pool-a47e39994ef3497e96bedd8a36133e00.run_cpu_time_us) (tablet-download-pool-a47e39994ef3497e96bedd8a36133e00.run_wall_time_us, tablet-download-pool-a47e39994ef3497e96bedd8a36133e00.run_wall_time_us) (tablet-download-pool-a726c9cea76241df8a00d52d6175db7a.queue_time_us, tablet-download-pool-a726c9cea76241df8a00d52d6175db7a.queue_time_us) (tablet-download-pool-a726c9cea76241df8a00d52d6175db7a.run_cpu_time_us, tablet-download-pool-a726c9cea76241df8a00d52d6175db7a.run_cpu_time_us) (tablet-download-pool-a726c9cea76241df8a00d52d6175db7a.run_wall_time_us, tablet-download-pool-a726c9cea76241df8a00d52d6175db7a.run_wall_time_us) (tablet-download-pool-b8922efb08b3491fa8d947793206bf7a.queue_time_us, tablet-download-pool-b8922efb08b3491fa8d947793206bf7a.queue_time_us) (tablet-download-pool-f796e2ca050e4666b9eb2dfd8a9ee2ff.queue_time_us, tablet-download-pool-f796e2ca050e4666b9eb2dfd8a9ee2ff.queue_time_us) (tablet-download-pool-f796e2ca050e4666b9eb2dfd8a9ee2ff.run_cpu_time_us, tablet-download-pool-f796e2ca050e4666b9eb2dfd8a9ee2ff.run_cpu_time_us) (tablet-download-pool-f796e2ca050e4666b9eb2dfd8a9ee2ff.run_wall_time_us, tablet-download-pool-f796e2ca050e4666b9eb2dfd8a9ee2ff.run_wall_time_us) (tablet-download-pool-ffd3ce2e5a1a4a0182e578d471a022ae.queue_time_us, tablet-download-pool-ffd3ce2e5a1a4a0182e578d471a022ae.queue_time_us) (tablet-download-pool-ffd3ce2e5a1a4a0182e578d471a022ae.run_cpu_time_us, tablet-download-pool-ffd3ce2e5a1a4a0182e578d471a022ae.run_cpu_time_us) (tablet-download-pool-ffd3ce2e5a1a4a0182e578d471a022ae.run_wall_time_us, tablet-download-pool-ffd3ce2e5a1a4a0182e578d471a022ae.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
*** Aborted at 1714572636 (unix time) try "date -d @1714572636" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800004037) received by PID 16439 (TID 0x7f1a281ae700) from PID 16439; stack trace: ***
@ 0x7f1a5b35a040 (unknown) at ??:0
@ 0x7f1a5b359fb7 gsignal at ??:0
@ 0x7f1a5b35b921 abort at ??:0
@ 0x7f1a5c05edcd google::LogMessage::Fail() at ??:0
@ 0x7f1a5c062b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f1a5c05e7cc google::LogMessage::Flush() at ??:0
@ 0x7f1a5c05ff59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1a5cbb6044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f1a5cb9af98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f1a5cb98efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f1a5db1577a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f1a5db1ea45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f1a5db9f76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f1a5dbb34f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f1a5db9e226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f1a5dbadc0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f1a5e1083cc std::function<>::operator()() at ??:0
@ 0x7f1a5cb9de06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1a5cb9e6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f1a5cba003b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f1a5e1083cc std::function<>::operator()() at ??:0
@ 0x7f1a5cb8d807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1a5938f6db start_thread at ??:0
@ 0x7f1a5b43c71f clone at ??:0