[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 20:45:26.540338 22946 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.22.104.190:41025,127.22.104.189:36603,127.22.104.188:38909
I20240502 20:45:26.541520 22946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:26.542163 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:26.549633 22954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:26.549952 22952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:26.550410 22951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:26.550838 22946 server_base.cc:1034] running on GCE node
I20240502 20:45:26.551349 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:45:26.551470 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:45:26.551518 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682726551515 us; error 0 us; skew 500 ppm
I20240502 20:45:26.557833 22946 webserver.cc:460] Webserver started at http://127.22.104.190:36547/ using document root <none> and password file <none>
I20240502 20:45:26.558393 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:26.558454 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:26.558673 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:26.560456 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714682726530369-22946-0/minicluster-data/master-0-root/instance:
uuid: "e5e02cc3547149ae890ef1eb4ebec379"
format_stamp: "Formatted at 2024-05-02 20:45:26 on dist-test-slave-rgns"
I20240502 20:45:26.563994 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240502 20:45:26.566092 22959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:26.567225 22946 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 20:45:26.567330 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714682726530369-22946-0/minicluster-data/master-0-root
uuid: "e5e02cc3547149ae890ef1eb4ebec379"
format_stamp: "Formatted at 2024-05-02 20:45:26 on dist-test-slave-rgns"
I20240502 20:45:26.567418 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714682726530369-22946-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714682726530369-22946-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714682726530369-22946-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:26.598490 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:26.599239 22946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:26.599406 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:26.606155 23012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.190:41025 every 8 connection(s)
I20240502 20:45:26.606155 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.190:41025
I20240502 20:45:26.607473 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:26.609643 23018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:26.609776 23013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 20:45:26.609851 23015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:26.610148 23016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:26.610502 22946 server_base.cc:1034] running on GCE node
I20240502 20:45:26.610667 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:45:26.610730 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:45:26.610754 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682726610752 us; error 0 us; skew 500 ppm
I20240502 20:45:26.611703 22946 webserver.cc:460] Webserver started at http://127.22.104.189:41891/ using document root <none> and password file <none>
I20240502 20:45:26.611853 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:26.611896 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:26.611968 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:26.612314 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714682726530369-22946-0/minicluster-data/master-1-root/instance:
uuid: "cd1573c66ef74f41bce4b6d7d30f0576"
format_stamp: "Formatted at 2024-05-02 20:45:26 on dist-test-slave-rgns"
I20240502 20:45:26.613947 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 20:45:26.614392 23013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.104.190" port: 41025 } has no permanent_uuid. Determining permanent_uuid...
I20240502 20:45:26.614851 23023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:26.615113 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:26.615191 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714682726530369-22946-0/minicluster-data/master-1-root
uuid: "cd1573c66ef74f41bce4b6d7d30f0576"
format_stamp: "Formatted at 2024-05-02 20:45:26 on dist-test-slave-rgns"
I20240502 20:45:26.615249 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714682726530369-22946-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714682726530369-22946-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714682726530369-22946-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:26.624536 23013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.104.189" port: 36603 } has no permanent_uuid. Determining permanent_uuid...
W20240502 20:45:26.625238 22960 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.104.189:36603: connect: Connection refused (error 111)
W20240502 20:45:26.626029 23013 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.104.189:36603: Network error: Client connection negotiation failed: client connection to 127.22.104.189:36603: connect: Connection refused (error 111)
I20240502 20:45:26.628198 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:26.628561 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:26.633111 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.189:36603
I20240502 20:45:26.633167 23079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.189:36603 every 8 connection(s)
I20240502 20:45:26.633774 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 20:45:26.633956 23080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:26.635222 23080 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.104.190" port: 41025 } has no permanent_uuid. Determining permanent_uuid...
W20240502 20:45:26.635700 23082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:26.636065 23083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:26.636734 23085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:26.637141 22946 server_base.cc:1034] running on GCE node
I20240502 20:45:26.637305 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:45:26.637354 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:45:26.637374 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682726637374 us; error 0 us; skew 500 ppm
I20240502 20:45:26.638365 22946 webserver.cc:460] Webserver started at http://127.22.104.188:38965/ using document root <none> and password file <none>
I20240502 20:45:26.638509 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:26.638554 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:26.638650 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:26.639091 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714682726530369-22946-0/minicluster-data/master-2-root/instance:
uuid: "bdc2896e260f472b884a650545f0a095"
format_stamp: "Formatted at 2024-05-02 20:45:26 on dist-test-slave-rgns"
I20240502 20:45:26.639420 23080 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.104.189" port: 36603 } has no permanent_uuid. Determining permanent_uuid...
I20240502 20:45:26.640663 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 20:45:26.641623 23092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:26.641933 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:26.642007 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714682726530369-22946-0/minicluster-data/master-2-root
uuid: "bdc2896e260f472b884a650545f0a095"
format_stamp: "Formatted at 2024-05-02 20:45:26 on dist-test-slave-rgns"
I20240502 20:45:26.642068 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714682726530369-22946-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714682726530369-22946-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714682726530369-22946-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:26.643337 23080 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.104.188" port: 38909 } has no permanent_uuid. Determining permanent_uuid...
W20240502 20:45:26.644493 23080 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.104.188:38909: Network error: Client connection negotiation failed: client connection to 127.22.104.188:38909: connect: Connection refused (error 111)
I20240502 20:45:26.659758 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:26.660202 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:26.664922 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.188:38909
I20240502 20:45:26.664986 23146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.188:38909 every 8 connection(s)
I20240502 20:45:26.665369 22946 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 20:45:26.665819 23147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:26.666960 23147 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.104.190" port: 41025 } has no permanent_uuid. Determining permanent_uuid...
I20240502 20:45:26.670886 23147 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.104.189" port: 36603 } has no permanent_uuid. Determining permanent_uuid...
I20240502 20:45:26.674134 23147 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.104.188" port: 38909 } has no permanent_uuid. Determining permanent_uuid...
I20240502 20:45:26.675220 23013 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.104.189" port: 36603 } attempt: 1
I20240502 20:45:26.678653 23013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.104.188" port: 38909 } has no permanent_uuid. Determining permanent_uuid...
I20240502 20:45:26.680307 23147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095: Bootstrap starting.
I20240502 20:45:26.682814 23013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5e02cc3547149ae890ef1eb4ebec379: Bootstrap starting.
I20240502 20:45:26.682929 23147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:26.683593 23013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5e02cc3547149ae890ef1eb4ebec379: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:26.683766 23147 log.cc:824] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:26.685257 23147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095: No bootstrap required, opened a new log
I20240502 20:45:26.685258 23013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5e02cc3547149ae890ef1eb4ebec379: No bootstrap required, opened a new log
I20240502 20:45:26.688297 23013 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5e02cc3547149ae890ef1eb4ebec379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e02cc3547149ae890ef1eb4ebec379" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 41025 } } peers { permanent_uuid: "cd1573c66ef74f41bce4b6d7d30f0576" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 36603 } } peers { permanent_uuid: "bdc2896e260f472b884a650545f0a095" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 38909 } }
I20240502 20:45:26.688287 23147 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e02cc3547149ae890ef1eb4ebec379" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 41025 } } peers { permanent_uuid: "cd1573c66ef74f41bce4b6d7d30f0576" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 36603 } } peers { permanent_uuid: "bdc2896e260f472b884a650545f0a095" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 38909 } }
I20240502 20:45:26.688454 23013 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5e02cc3547149ae890ef1eb4ebec379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:26.688480 23147 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:26.688488 23013 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5e02cc3547149ae890ef1eb4ebec379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e02cc3547149ae890ef1eb4ebec379, State: Initialized, Role: FOLLOWER
I20240502 20:45:26.688532 23147 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdc2896e260f472b884a650545f0a095, State: Initialized, Role: FOLLOWER
I20240502 20:45:26.688896 23147 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e02cc3547149ae890ef1eb4ebec379" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 41025 } } peers { permanent_uuid: "cd1573c66ef74f41bce4b6d7d30f0576" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 36603 } } peers { permanent_uuid: "bdc2896e260f472b884a650545f0a095" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 38909 } }
I20240502 20:45:26.688897 23013 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5e02cc3547149ae890ef1eb4ebec379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e02cc3547149ae890ef1eb4ebec379" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 41025 } } peers { permanent_uuid: "cd1573c66ef74f41bce4b6d7d30f0576" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 36603 } } peers { permanent_uuid: "bdc2896e260f472b884a650545f0a095" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 38909 } }
I20240502 20:45:26.689404 23154 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5e02cc3547149ae890ef1eb4ebec379 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e02cc3547149ae890ef1eb4ebec379" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 41025 } } peers { permanent_uuid: "cd1573c66ef74f41bce4b6d7d30f0576" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 36603 } } peers { permanent_uuid: "bdc2896e260f472b884a650545f0a095" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 38909 } } }
I20240502 20:45:26.689409 23153 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e02cc3547149ae890ef1eb4ebec379" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 41025 } } peers { permanent_uuid: "cd1573c66ef74f41bce4b6d7d30f0576" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 36603 } } peers { permanent_uuid: "bdc2896e260f472b884a650545f0a095" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 38909 } } }
I20240502 20:45:26.689635 23154 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5e02cc3547149ae890ef1eb4ebec379 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 20:45:26.689739 23153 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 20:45:26.690128 23013 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5e02cc3547149ae890ef1eb4ebec379 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 20:45:26.690132 23147 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 20:45:26.692147 22946 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20240502 20:45:26.694576 23175 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 20:45:26.694576 23176 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e5e02cc3547149ae890ef1eb4ebec379: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 20:45:26.694671 23175 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 20:45:26.694684 23176 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 20:45:26.696636 23080 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.104.188" port: 38909 } attempt: 1
I20240502 20:45:26.701309 23080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd1573c66ef74f41bce4b6d7d30f0576: Bootstrap starting.
I20240502 20:45:26.702302 23080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd1573c66ef74f41bce4b6d7d30f0576: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:26.703481 23080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd1573c66ef74f41bce4b6d7d30f0576: No bootstrap required, opened a new log
I20240502 20:45:26.703886 23080 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd1573c66ef74f41bce4b6d7d30f0576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e02cc3547149ae890ef1eb4ebec379" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 41025 } } peers { permanent_uuid: "cd1573c66ef74f41bce4b6d7d30f0576" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 36603 } } peers { permanent_uuid: "bdc2896e260f472b884a650545f0a095" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 38909 } }
I20240502 20:45:26.703986 23080 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd1573c66ef74f41bce4b6d7d30f0576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:26.704017 23080 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd1573c66ef74f41bce4b6d7d30f0576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd1573c66ef74f41bce4b6d7d30f0576, State: Initialized, Role: FOLLOWER
I20240502 20:45:26.704152 23080 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd1573c66ef74f41bce4b6d7d30f0576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e02cc3547149ae890ef1eb4ebec379" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 41025 } } peers { permanent_uuid: "cd1573c66ef74f41bce4b6d7d30f0576" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 36603 } } peers { permanent_uuid: "bdc2896e260f472b884a650545f0a095" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 38909 } }
I20240502 20:45:26.704473 23179 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd1573c66ef74f41bce4b6d7d30f0576 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e02cc3547149ae890ef1eb4ebec379" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 41025 } } peers { permanent_uuid: "cd1573c66ef74f41bce4b6d7d30f0576" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 36603 } } peers { permanent_uuid: "bdc2896e260f472b884a650545f0a095" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 38909 } } }
I20240502 20:45:26.704584 23080 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd1573c66ef74f41bce4b6d7d30f0576 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 20:45:26.704612 23179 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd1573c66ef74f41bce4b6d7d30f0576 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 20:45:26.706282 22946 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20240502 20:45:26.706884 23190 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cd1573c66ef74f41bce4b6d7d30f0576: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 20:45:26.706940 23190 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 20:45:26.706959 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:26.709471 23192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:26.709848 23191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:26.711851 23194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:26.712213 22946 server_base.cc:1034] running on GCE node
I20240502 20:45:26.712380 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:45:26.712425 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:45:26.712446 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682726712445 us; error 0 us; skew 500 ppm
I20240502 20:45:26.713508 22946 webserver.cc:460] Webserver started at http://127.22.104.129:39225/ using document root <none> and password file <none>
I20240502 20:45:26.713670 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:26.713722 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:26.713793 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:26.714239 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714682726530369-22946-0/minicluster-data/ts-0-root/instance:
uuid: "b10d3d589810446888fbbb558b3d1b03"
format_stamp: "Formatted at 2024-05-02 20:45:26 on dist-test-slave-rgns"
I20240502 20:45:26.715989 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240502 20:45:26.716989 23199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:26.717288 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 20:45:26.717350 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714682726530369-22946-0/minicluster-data/ts-0-root
uuid: "b10d3d589810446888fbbb558b3d1b03"
format_stamp: "Formatted at 2024-05-02 20:45:26 on dist-test-slave-rgns"
I20240502 20:45:26.717403 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714682726530369-22946-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714682726530369-22946-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714682726530369-22946-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:26.732542 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:26.733023 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:26.733520 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:26.734391 22946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:26.734457 22946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:26.734509 22946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:26.734542 22946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:26.740597 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.129:37329
I20240502 20:45:26.740644 23262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.129:37329 every 8 connection(s)
I20240502 20:45:26.741700 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:26.743985 23270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:26.747253 23264 heartbeater.cc:347] Connected to a master server at 127.22.104.190:41025
I20240502 20:45:26.747388 23264 heartbeater.cc:464] Registering TS with master...
W20240502 20:45:26.747678 23273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:26.747892 23271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:26.748035 23264 heartbeater.cc:511] Master 127.22.104.190:41025 requested a full tablet report, sending...
I20240502 20:45:26.748585 22946 server_base.cc:1034] running on GCE node
I20240502 20:45:26.748726 23265 heartbeater.cc:347] Connected to a master server at 127.22.104.189:36603
I20240502 20:45:26.748723 23263 heartbeater.cc:347] Connected to a master server at 127.22.104.188:38909
I20240502 20:45:26.748786 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:45:26.748824 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:45:26.748829 23265 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:26.748857 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682726748856 us; error 0 us; skew 500 ppm
I20240502 20:45:26.749074 23265 heartbeater.cc:511] Master 127.22.104.189:36603 requested a full tablet report, sending...
I20240502 20:45:26.748829 23263 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:26.749478 23263 heartbeater.cc:511] Master 127.22.104.188:38909 requested a full tablet report, sending...
I20240502 20:45:26.749801 22976 ts_manager.cc:194] Registered new tserver with Master: b10d3d589810446888fbbb558b3d1b03 (127.22.104.129:37329)
I20240502 20:45:26.749944 22946 webserver.cc:460] Webserver started at http://127.22.104.130:33173/ using document root <none> and password file <none>
I20240502 20:45:26.750088 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:26.750140 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:26.750172 23110 ts_manager.cc:194] Registered new tserver with Master: b10d3d589810446888fbbb558b3d1b03 (127.22.104.129:37329)
I20240502 20:45:26.750207 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:26.750670 23044 ts_manager.cc:194] Registered new tserver with Master: b10d3d589810446888fbbb558b3d1b03 (127.22.104.129:37329)
I20240502 20:45:26.750674 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714682726530369-22946-0/minicluster-data/ts-1-root/instance:
uuid: "6435a7db961643ba80a278390d0d97ec"
format_stamp: "Formatted at 2024-05-02 20:45:26 on dist-test-slave-rgns"
I20240502 20:45:26.752234 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:26.753120 23278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:26.753403 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:26.753487 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714682726530369-22946-0/minicluster-data/ts-1-root
uuid: "6435a7db961643ba80a278390d0d97ec"
format_stamp: "Formatted at 2024-05-02 20:45:26 on dist-test-slave-rgns"
I20240502 20:45:26.753569 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714682726530369-22946-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714682726530369-22946-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714682726530369-22946-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:26.777611 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:26.778071 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:26.778409 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:26.778987 22946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:26.779031 22946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:26.779064 22946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:26.779088 22946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:26.785306 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.130:38291
I20240502 20:45:26.785375 23342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.130:38291 every 8 connection(s)
I20240502 20:45:26.786283 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:26.791471 23349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:26.791580 23344 heartbeater.cc:347] Connected to a master server at 127.22.104.190:41025
I20240502 20:45:26.791493 23343 heartbeater.cc:347] Connected to a master server at 127.22.104.188:38909
I20240502 20:45:26.791673 23344 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:26.791704 23343 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:26.791921 23343 heartbeater.cc:511] Master 127.22.104.188:38909 requested a full tablet report, sending...
W20240502 20:45:26.792069 23350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:26.792282 23344 heartbeater.cc:511] Master 127.22.104.190:41025 requested a full tablet report, sending...
W20240502 20:45:26.792516 23352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:26.792627 23110 ts_manager.cc:194] Registered new tserver with Master: 6435a7db961643ba80a278390d0d97ec (127.22.104.130:38291)
I20240502 20:45:26.792757 22946 server_base.cc:1034] running on GCE node
I20240502 20:45:26.793031 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:45:26.793097 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:45:26.793146 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682726793145 us; error 0 us; skew 500 ppm
I20240502 20:45:26.793308 22976 ts_manager.cc:194] Registered new tserver with Master: 6435a7db961643ba80a278390d0d97ec (127.22.104.130:38291)
I20240502 20:45:26.794358 22946 webserver.cc:460] Webserver started at http://127.22.104.131:45107/ using document root <none> and password file <none>
I20240502 20:45:26.794529 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:26.794597 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:26.794667 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:26.795150 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714682726530369-22946-0/minicluster-data/ts-2-root/instance:
uuid: "423887a1c6f84a4f84b53bf6c5052023"
format_stamp: "Formatted at 2024-05-02 20:45:26 on dist-test-slave-rgns"
I20240502 20:45:26.795456 23345 heartbeater.cc:347] Connected to a master server at 127.22.104.189:36603
I20240502 20:45:26.795540 23345 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:26.795753 23345 heartbeater.cc:511] Master 127.22.104.189:36603 requested a full tablet report, sending...
I20240502 20:45:26.796402 23044 ts_manager.cc:194] Registered new tserver with Master: 6435a7db961643ba80a278390d0d97ec (127.22.104.130:38291)
I20240502 20:45:26.796949 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240502 20:45:26.797883 23358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:26.798175 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 20:45:26.798243 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714682726530369-22946-0/minicluster-data/ts-2-root
uuid: "423887a1c6f84a4f84b53bf6c5052023"
format_stamp: "Formatted at 2024-05-02 20:45:26 on dist-test-slave-rgns"
I20240502 20:45:26.798303 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714682726530369-22946-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714682726530369-22946-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714682726530369-22946-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:26.800969 23153 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:26.801131 23153 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e02cc3547149ae890ef1eb4ebec379" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 41025 } } peers { permanent_uuid: "cd1573c66ef74f41bce4b6d7d30f0576" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 36603 } } peers { permanent_uuid: "bdc2896e260f472b884a650545f0a095" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 38909 } }
I20240502 20:45:26.801980 23153 leader_election.cc:288] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5e02cc3547149ae890ef1eb4ebec379 (127.22.104.190:41025), cd1573c66ef74f41bce4b6d7d30f0576 (127.22.104.189:36603)
I20240502 20:45:26.802512 22986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bdc2896e260f472b884a650545f0a095" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e02cc3547149ae890ef1eb4ebec379" is_pre_election: true
I20240502 20:45:26.802688 22986 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e5e02cc3547149ae890ef1eb4ebec379 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdc2896e260f472b884a650545f0a095 in term 0.
I20240502 20:45:26.802937 23054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bdc2896e260f472b884a650545f0a095" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd1573c66ef74f41bce4b6d7d30f0576" is_pre_election: true
I20240502 20:45:26.803022 23054 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cd1573c66ef74f41bce4b6d7d30f0576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdc2896e260f472b884a650545f0a095 in term 0.
I20240502 20:45:26.803067 23093 leader_election.cc:302] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdc2896e260f472b884a650545f0a095, e5e02cc3547149ae890ef1eb4ebec379; no voters:
I20240502 20:45:26.803464 23153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:26.803530 23153 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:26.803563 23153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:26.804490 23153 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e02cc3547149ae890ef1eb4ebec379" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 41025 } } peers { permanent_uuid: "cd1573c66ef74f41bce4b6d7d30f0576" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 36603 } } peers { permanent_uuid: "bdc2896e260f472b884a650545f0a095" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 38909 } }
I20240502 20:45:26.804803 23153 leader_election.cc:288] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095 [CANDIDATE]: Term 1 election: Requested vote from peers e5e02cc3547149ae890ef1eb4ebec379 (127.22.104.190:41025), cd1573c66ef74f41bce4b6d7d30f0576 (127.22.104.189:36603)
I20240502 20:45:26.805210 23054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bdc2896e260f472b884a650545f0a095" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd1573c66ef74f41bce4b6d7d30f0576"
I20240502 20:45:26.805302 23054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd1573c66ef74f41bce4b6d7d30f0576 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:26.806102 23054 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cd1573c66ef74f41bce4b6d7d30f0576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdc2896e260f472b884a650545f0a095 in term 1.
I20240502 20:45:26.806383 23093 leader_election.cc:302] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdc2896e260f472b884a650545f0a095, cd1573c66ef74f41bce4b6d7d30f0576; no voters:
I20240502 20:45:26.806470 22986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bdc2896e260f472b884a650545f0a095" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e02cc3547149ae890ef1eb4ebec379"
I20240502 20:45:26.806555 22986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5e02cc3547149ae890ef1eb4ebec379 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:26.806766 23153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:26.806951 23153 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095 [term 1 LEADER]: Becoming Leader. State: Replica: bdc2896e260f472b884a650545f0a095, State: Running, Role: LEADER
I20240502 20:45:26.807216 23153 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e02cc3547149ae890ef1eb4ebec379" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 41025 } } peers { permanent_uuid: "cd1573c66ef74f41bce4b6d7d30f0576" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 36603 } } peers { permanent_uuid: "bdc2896e260f472b884a650545f0a095" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 38909 } }
I20240502 20:45:26.807318 22986 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e5e02cc3547149ae890ef1eb4ebec379 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdc2896e260f472b884a650545f0a095 in term 1.
I20240502 20:45:26.808806 23153 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdc2896e260f472b884a650545f0a095. Latest consensus state: current_term: 1 leader_uuid: "bdc2896e260f472b884a650545f0a095" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e02cc3547149ae890ef1eb4ebec379" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 41025 } } peers { permanent_uuid: "cd1573c66ef74f41bce4b6d7d30f0576" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 36603 } } peers { permanent_uuid: "bdc2896e260f472b884a650545f0a095" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 38909 } } }
I20240502 20:45:26.808914 23153 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095 [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:26.809267 23365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 20:45:26.810313 23365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 20:45:26.810470 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:26.810906 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:26.811197 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:26.811691 22946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:26.811731 22946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:26.811774 22946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:26.811802 22946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:26.816226 23054 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cd1573c66ef74f41bce4b6d7d30f0576 [term 1 FOLLOWER]: Refusing update from remote peer bdc2896e260f472b884a650545f0a095: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:26.816723 22986 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e5e02cc3547149ae890ef1eb4ebec379 [term 1 FOLLOWER]: Refusing update from remote peer bdc2896e260f472b884a650545f0a095: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:26.817252 23363 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd1573c66ef74f41bce4b6d7d30f0576" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 36603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:26.817662 23153 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e02cc3547149ae890ef1eb4ebec379" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 41025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:26.817796 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.131:34059
I20240502 20:45:26.818787 23426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.131:34059 every 8 connection(s)
I20240502 20:45:26.819207 23179 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd1573c66ef74f41bce4b6d7d30f0576 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdc2896e260f472b884a650545f0a095. Latest consensus state: current_term: 1 leader_uuid: "bdc2896e260f472b884a650545f0a095" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e02cc3547149ae890ef1eb4ebec379" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 41025 } } peers { permanent_uuid: "cd1573c66ef74f41bce4b6d7d30f0576" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 36603 } } peers { permanent_uuid: "bdc2896e260f472b884a650545f0a095" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 38909 } } }
I20240502 20:45:26.819325 23179 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd1573c66ef74f41bce4b6d7d30f0576 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 20:45:26.820466 23154 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5e02cc3547149ae890ef1eb4ebec379 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdc2896e260f472b884a650545f0a095. Latest consensus state: current_term: 1 leader_uuid: "bdc2896e260f472b884a650545f0a095" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e02cc3547149ae890ef1eb4ebec379" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 41025 } } peers { permanent_uuid: "cd1573c66ef74f41bce4b6d7d30f0576" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 36603 } } peers { permanent_uuid: "bdc2896e260f472b884a650545f0a095" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 38909 } } }
I20240502 20:45:26.820569 23154 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5e02cc3547149ae890ef1eb4ebec379 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 20:45:26.821241 23153 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bdc2896e260f472b884a650545f0a095" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e02cc3547149ae890ef1eb4ebec379" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 41025 } } peers { permanent_uuid: "cd1573c66ef74f41bce4b6d7d30f0576" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 36603 } } peers { permanent_uuid: "bdc2896e260f472b884a650545f0a095" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 38909 } } }
I20240502 20:45:26.821342 23153 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095 [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:26.821444 23369 mvcc.cc:205] Tried to move back new op lower bound from 7023340449030967296 to 7023340449005268992. Current Snapshot: MvccSnapshot[applied={T|T < 7023340449030967296}]
I20240502 20:45:26.822335 23179 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd1573c66ef74f41bce4b6d7d30f0576 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bdc2896e260f472b884a650545f0a095" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e02cc3547149ae890ef1eb4ebec379" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 41025 } } peers { permanent_uuid: "cd1573c66ef74f41bce4b6d7d30f0576" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 36603 } } peers { permanent_uuid: "bdc2896e260f472b884a650545f0a095" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 38909 } } }
I20240502 20:45:26.822427 23179 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd1573c66ef74f41bce4b6d7d30f0576 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 20:45:26.824939 23427 heartbeater.cc:347] Connected to a master server at 127.22.104.188:38909
I20240502 20:45:26.825026 23427 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:26.825237 23427 heartbeater.cc:511] Master 127.22.104.188:38909 requested a full tablet report, sending...
I20240502 20:45:26.826009 23110 ts_manager.cc:194] Registered new tserver with Master: 423887a1c6f84a4f84b53bf6c5052023 (127.22.104.131:34059)
I20240502 20:45:26.826406 23153 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bdc2896e260f472b884a650545f0a095" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e02cc3547149ae890ef1eb4ebec379" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 41025 } } peers { permanent_uuid: "cd1573c66ef74f41bce4b6d7d30f0576" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 36603 } } peers { permanent_uuid: "bdc2896e260f472b884a650545f0a095" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 38909 } } }
I20240502 20:45:26.826472 23428 heartbeater.cc:347] Connected to a master server at 127.22.104.190:41025
I20240502 20:45:26.826509 23153 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095 [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:26.826541 23428 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:26.826750 23428 heartbeater.cc:511] Master 127.22.104.190:41025 requested a full tablet report, sending...
I20240502 20:45:26.827219 23365 catalog_manager.cc:1347] Generated new cluster ID: b632d98c4bd54accbdd698704bf4ee24
I20240502 20:45:26.827271 23365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 20:45:26.827543 23154 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5e02cc3547149ae890ef1eb4ebec379 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bdc2896e260f472b884a650545f0a095" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e02cc3547149ae890ef1eb4ebec379" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 41025 } } peers { permanent_uuid: "cd1573c66ef74f41bce4b6d7d30f0576" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 36603 } } peers { permanent_uuid: "bdc2896e260f472b884a650545f0a095" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 38909 } } }
I20240502 20:45:26.827632 23154 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5e02cc3547149ae890ef1eb4ebec379 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 20:45:26.831192 22976 ts_manager.cc:194] Registered new tserver with Master: 423887a1c6f84a4f84b53bf6c5052023 (127.22.104.131:34059)
I20240502 20:45:26.831476 23429 heartbeater.cc:347] Connected to a master server at 127.22.104.189:36603
I20240502 20:45:26.831565 23429 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:26.831789 23429 heartbeater.cc:511] Master 127.22.104.189:36603 requested a full tablet report, sending...
I20240502 20:45:26.832369 23044 ts_manager.cc:194] Registered new tserver with Master: 423887a1c6f84a4f84b53bf6c5052023 (127.22.104.131:34059)
I20240502 20:45:26.832832 22946 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014479002s
I20240502 20:45:26.845125 23365 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 20:45:26.846276 23365 catalog_manager.cc:1504] Loading token signing keys...
I20240502 20:45:26.859122 23365 catalog_manager.cc:5857] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095: Generated new TSK 0
I20240502 20:45:26.859944 23365 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 20:45:27.695710 23176 catalog_manager.cc:1259] Loaded cluster ID: b632d98c4bd54accbdd698704bf4ee24
I20240502 20:45:27.695796 23176 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e5e02cc3547149ae890ef1eb4ebec379: loading cluster ID for follower catalog manager: success
I20240502 20:45:27.697082 23176 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e5e02cc3547149ae890ef1eb4ebec379: acquiring CA information for follower catalog manager: success
I20240502 20:45:27.697742 23176 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e5e02cc3547149ae890ef1eb4ebec379: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 20:45:27.707648 23190 catalog_manager.cc:1259] Loaded cluster ID: b632d98c4bd54accbdd698704bf4ee24
I20240502 20:45:27.707723 23190 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cd1573c66ef74f41bce4b6d7d30f0576: loading cluster ID for follower catalog manager: success
I20240502 20:45:27.708840 23190 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cd1573c66ef74f41bce4b6d7d30f0576: acquiring CA information for follower catalog manager: success
I20240502 20:45:27.709335 23190 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cd1573c66ef74f41bce4b6d7d30f0576: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 20:45:27.752985 23110 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49704
I20240502 20:45:27.794859 23110 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49708
I20240502 20:45:27.832643 23110 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49716
I20240502 20:45:28.692277 23174 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:45:28.754303 23263 heartbeater.cc:503] Master 127.22.104.188:38909 was elected leader, sending a full tablet report...
I20240502 20:45:28.795753 23343 heartbeater.cc:503] Master 127.22.104.188:38909 was elected leader, sending a full tablet report...
I20240502 20:45:28.833658 23427 heartbeater.cc:503] Master 127.22.104.188:38909 was elected leader, sending a full tablet report...
I20240502 20:45:28.987699 22946 test_util.cc:268] Using random seed: -1359561685
I20240502 20:45:29.001152 23110 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49724:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 20:45:29.019912 23226 tablet_service.cc:1467] Processing CreateTablet for tablet 0d13aad3a9eb4ffab2f7c4480661175c (DEFAULT_TABLE table=test-workload [id=9ca8becd1bbf409b94a5e726ab6d5c4c]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 20:45:29.020117 23391 tablet_service.cc:1467] Processing CreateTablet for tablet 2337eab6e13f401b9f05e5b760ce1d71 (DEFAULT_TABLE table=test-workload [id=9ca8becd1bbf409b94a5e726ab6d5c4c]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 20:45:29.020117 23307 tablet_service.cc:1467] Processing CreateTablet for tablet b1adcabb2b7d494fb44469769285ad43 (DEFAULT_TABLE table=test-workload [id=9ca8becd1bbf409b94a5e726ab6d5c4c]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 20:45:29.020408 23226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d13aad3a9eb4ffab2f7c4480661175c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:29.020432 23391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2337eab6e13f401b9f05e5b760ce1d71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:29.020449 23307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1adcabb2b7d494fb44469769285ad43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:29.019912 23227 tablet_service.cc:1467] Processing CreateTablet for tablet 5b16516c947042aab223907ea293b3ef (DEFAULT_TABLE table=test-workload [id=9ca8becd1bbf409b94a5e726ab6d5c4c]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 20:45:29.022958 23471 tablet_bootstrap.cc:492] T 0d13aad3a9eb4ffab2f7c4480661175c P b10d3d589810446888fbbb558b3d1b03: Bootstrap starting.
I20240502 20:45:29.023131 23227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b16516c947042aab223907ea293b3ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:29.023792 23471 tablet_bootstrap.cc:654] T 0d13aad3a9eb4ffab2f7c4480661175c P b10d3d589810446888fbbb558b3d1b03: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:29.024689 23473 tablet_bootstrap.cc:492] T 2337eab6e13f401b9f05e5b760ce1d71 P 423887a1c6f84a4f84b53bf6c5052023: Bootstrap starting.
I20240502 20:45:29.025116 23471 tablet_bootstrap.cc:492] T 0d13aad3a9eb4ffab2f7c4480661175c P b10d3d589810446888fbbb558b3d1b03: No bootstrap required, opened a new log
I20240502 20:45:29.025216 23471 ts_tablet_manager.cc:1367] T 0d13aad3a9eb4ffab2f7c4480661175c P b10d3d589810446888fbbb558b3d1b03: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:45:29.025389 23473 tablet_bootstrap.cc:654] T 2337eab6e13f401b9f05e5b760ce1d71 P 423887a1c6f84a4f84b53bf6c5052023: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:29.025574 23471 raft_consensus.cc:348] T 0d13aad3a9eb4ffab2f7c4480661175c P b10d3d589810446888fbbb558b3d1b03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b10d3d589810446888fbbb558b3d1b03" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 37329 } }
I20240502 20:45:29.025660 23471 raft_consensus.cc:374] T 0d13aad3a9eb4ffab2f7c4480661175c P b10d3d589810446888fbbb558b3d1b03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:29.025681 23471 raft_consensus.cc:729] T 0d13aad3a9eb4ffab2f7c4480661175c P b10d3d589810446888fbbb558b3d1b03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b10d3d589810446888fbbb558b3d1b03, State: Initialized, Role: FOLLOWER
I20240502 20:45:29.025789 23471 consensus_queue.cc:260] T 0d13aad3a9eb4ffab2f7c4480661175c P b10d3d589810446888fbbb558b3d1b03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b10d3d589810446888fbbb558b3d1b03" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 37329 } }
I20240502 20:45:29.025871 23471 raft_consensus.cc:388] T 0d13aad3a9eb4ffab2f7c4480661175c P b10d3d589810446888fbbb558b3d1b03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:29.025919 23471 raft_consensus.cc:482] T 0d13aad3a9eb4ffab2f7c4480661175c P b10d3d589810446888fbbb558b3d1b03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:29.025970 23471 raft_consensus.cc:3037] T 0d13aad3a9eb4ffab2f7c4480661175c P b10d3d589810446888fbbb558b3d1b03 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:29.026070 23472 tablet_bootstrap.cc:492] T b1adcabb2b7d494fb44469769285ad43 P 6435a7db961643ba80a278390d0d97ec: Bootstrap starting.
I20240502 20:45:29.026414 23473 tablet_bootstrap.cc:492] T 2337eab6e13f401b9f05e5b760ce1d71 P 423887a1c6f84a4f84b53bf6c5052023: No bootstrap required, opened a new log
I20240502 20:45:29.026557 23473 ts_tablet_manager.cc:1367] T 2337eab6e13f401b9f05e5b760ce1d71 P 423887a1c6f84a4f84b53bf6c5052023: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:29.026569 23471 raft_consensus.cc:504] T 0d13aad3a9eb4ffab2f7c4480661175c P b10d3d589810446888fbbb558b3d1b03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b10d3d589810446888fbbb558b3d1b03" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 37329 } }
I20240502 20:45:29.026746 23471 leader_election.cc:302] T 0d13aad3a9eb4ffab2f7c4480661175c P b10d3d589810446888fbbb558b3d1b03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b10d3d589810446888fbbb558b3d1b03; no voters:
I20240502 20:45:29.026882 23472 tablet_bootstrap.cc:654] T b1adcabb2b7d494fb44469769285ad43 P 6435a7db961643ba80a278390d0d97ec: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:29.026908 23471 leader_election.cc:288] T 0d13aad3a9eb4ffab2f7c4480661175c P b10d3d589810446888fbbb558b3d1b03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:29.027006 23476 raft_consensus.cc:2781] T 0d13aad3a9eb4ffab2f7c4480661175c P b10d3d589810446888fbbb558b3d1b03 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:29.027172 23476 raft_consensus.cc:686] T 0d13aad3a9eb4ffab2f7c4480661175c P b10d3d589810446888fbbb558b3d1b03 [term 1 LEADER]: Becoming Leader. State: Replica: b10d3d589810446888fbbb558b3d1b03, State: Running, Role: LEADER
I20240502 20:45:29.027228 23471 ts_tablet_manager.cc:1398] T 0d13aad3a9eb4ffab2f7c4480661175c P b10d3d589810446888fbbb558b3d1b03: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:29.027297 23476 consensus_queue.cc:237] T 0d13aad3a9eb4ffab2f7c4480661175c P b10d3d589810446888fbbb558b3d1b03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b10d3d589810446888fbbb558b3d1b03" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 37329 } }
I20240502 20:45:29.027345 23473 raft_consensus.cc:348] T 2337eab6e13f401b9f05e5b760ce1d71 P 423887a1c6f84a4f84b53bf6c5052023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "423887a1c6f84a4f84b53bf6c5052023" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 34059 } }
I20240502 20:45:29.028504 23471 tablet_bootstrap.cc:492] T 5b16516c947042aab223907ea293b3ef P b10d3d589810446888fbbb558b3d1b03: Bootstrap starting.
I20240502 20:45:29.027433 23473 raft_consensus.cc:374] T 2337eab6e13f401b9f05e5b760ce1d71 P 423887a1c6f84a4f84b53bf6c5052023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:29.028868 23473 raft_consensus.cc:729] T 2337eab6e13f401b9f05e5b760ce1d71 P 423887a1c6f84a4f84b53bf6c5052023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 423887a1c6f84a4f84b53bf6c5052023, State: Initialized, Role: FOLLOWER
I20240502 20:45:29.029009 23473 consensus_queue.cc:260] T 2337eab6e13f401b9f05e5b760ce1d71 P 423887a1c6f84a4f84b53bf6c5052023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "423887a1c6f84a4f84b53bf6c5052023" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 34059 } }
I20240502 20:45:29.029104 23473 raft_consensus.cc:388] T 2337eab6e13f401b9f05e5b760ce1d71 P 423887a1c6f84a4f84b53bf6c5052023 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:29.029136 23473 raft_consensus.cc:482] T 2337eab6e13f401b9f05e5b760ce1d71 P 423887a1c6f84a4f84b53bf6c5052023 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:29.029711 23471 tablet_bootstrap.cc:654] T 5b16516c947042aab223907ea293b3ef P b10d3d589810446888fbbb558b3d1b03: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:29.029565 23110 catalog_manager.cc:5484] T 0d13aad3a9eb4ffab2f7c4480661175c P b10d3d589810446888fbbb558b3d1b03 reported cstate change: term changed from 0 to 1, leader changed from <none> to b10d3d589810446888fbbb558b3d1b03 (127.22.104.129). New cstate: current_term: 1 leader_uuid: "b10d3d589810446888fbbb558b3d1b03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b10d3d589810446888fbbb558b3d1b03" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 37329 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:29.029171 23473 raft_consensus.cc:3037] T 2337eab6e13f401b9f05e5b760ce1d71 P 423887a1c6f84a4f84b53bf6c5052023 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:29.031198 23471 tablet_bootstrap.cc:492] T 5b16516c947042aab223907ea293b3ef P b10d3d589810446888fbbb558b3d1b03: No bootstrap required, opened a new log
I20240502 20:45:29.031278 23471 ts_tablet_manager.cc:1367] T 5b16516c947042aab223907ea293b3ef P b10d3d589810446888fbbb558b3d1b03: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:45:29.031646 23471 raft_consensus.cc:348] T 5b16516c947042aab223907ea293b3ef P b10d3d589810446888fbbb558b3d1b03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b10d3d589810446888fbbb558b3d1b03" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 37329 } }
I20240502 20:45:29.031728 23471 raft_consensus.cc:374] T 5b16516c947042aab223907ea293b3ef P b10d3d589810446888fbbb558b3d1b03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:29.031757 23471 raft_consensus.cc:729] T 5b16516c947042aab223907ea293b3ef P b10d3d589810446888fbbb558b3d1b03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b10d3d589810446888fbbb558b3d1b03, State: Initialized, Role: FOLLOWER
I20240502 20:45:29.031877 23471 consensus_queue.cc:260] T 5b16516c947042aab223907ea293b3ef P b10d3d589810446888fbbb558b3d1b03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b10d3d589810446888fbbb558b3d1b03" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 37329 } }
I20240502 20:45:29.031970 23471 raft_consensus.cc:388] T 5b16516c947042aab223907ea293b3ef P b10d3d589810446888fbbb558b3d1b03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:29.032002 23471 raft_consensus.cc:482] T 5b16516c947042aab223907ea293b3ef P b10d3d589810446888fbbb558b3d1b03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:29.032037 23471 raft_consensus.cc:3037] T 5b16516c947042aab223907ea293b3ef P b10d3d589810446888fbbb558b3d1b03 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:29.032095 23473 raft_consensus.cc:504] T 2337eab6e13f401b9f05e5b760ce1d71 P 423887a1c6f84a4f84b53bf6c5052023 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "423887a1c6f84a4f84b53bf6c5052023" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 34059 } }
I20240502 20:45:29.032212 23473 leader_election.cc:302] T 2337eab6e13f401b9f05e5b760ce1d71 P 423887a1c6f84a4f84b53bf6c5052023 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 423887a1c6f84a4f84b53bf6c5052023; no voters:
I20240502 20:45:29.032375 23473 leader_election.cc:288] T 2337eab6e13f401b9f05e5b760ce1d71 P 423887a1c6f84a4f84b53bf6c5052023 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:29.032662 23473 ts_tablet_manager.cc:1398] T 2337eab6e13f401b9f05e5b760ce1d71 P 423887a1c6f84a4f84b53bf6c5052023: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 20:45:29.033929 23471 raft_consensus.cc:504] T 5b16516c947042aab223907ea293b3ef P b10d3d589810446888fbbb558b3d1b03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b10d3d589810446888fbbb558b3d1b03" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 37329 } }
I20240502 20:45:29.034051 23471 leader_election.cc:302] T 5b16516c947042aab223907ea293b3ef P b10d3d589810446888fbbb558b3d1b03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b10d3d589810446888fbbb558b3d1b03; no voters:
I20240502 20:45:29.034145 23471 leader_election.cc:288] T 5b16516c947042aab223907ea293b3ef P b10d3d589810446888fbbb558b3d1b03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:29.034349 23471 ts_tablet_manager.cc:1398] T 5b16516c947042aab223907ea293b3ef P b10d3d589810446888fbbb558b3d1b03: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:45:29.034452 23476 raft_consensus.cc:2781] T 5b16516c947042aab223907ea293b3ef P b10d3d589810446888fbbb558b3d1b03 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:29.034518 23476 raft_consensus.cc:686] T 5b16516c947042aab223907ea293b3ef P b10d3d589810446888fbbb558b3d1b03 [term 1 LEADER]: Becoming Leader. State: Replica: b10d3d589810446888fbbb558b3d1b03, State: Running, Role: LEADER
I20240502 20:45:29.034612 23476 consensus_queue.cc:237] T 5b16516c947042aab223907ea293b3ef P b10d3d589810446888fbbb558b3d1b03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b10d3d589810446888fbbb558b3d1b03" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 37329 } }
I20240502 20:45:29.035421 23472 tablet_bootstrap.cc:492] T b1adcabb2b7d494fb44469769285ad43 P 6435a7db961643ba80a278390d0d97ec: No bootstrap required, opened a new log
I20240502 20:45:29.035501 23472 ts_tablet_manager.cc:1367] T b1adcabb2b7d494fb44469769285ad43 P 6435a7db961643ba80a278390d0d97ec: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 20:45:29.035874 23472 raft_consensus.cc:348] T b1adcabb2b7d494fb44469769285ad43 P 6435a7db961643ba80a278390d0d97ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6435a7db961643ba80a278390d0d97ec" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 38291 } }
I20240502 20:45:29.035949 23472 raft_consensus.cc:374] T b1adcabb2b7d494fb44469769285ad43 P 6435a7db961643ba80a278390d0d97ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:29.035977 23472 raft_consensus.cc:729] T b1adcabb2b7d494fb44469769285ad43 P 6435a7db961643ba80a278390d0d97ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6435a7db961643ba80a278390d0d97ec, State: Initialized, Role: FOLLOWER
I20240502 20:45:29.036073 23472 consensus_queue.cc:260] T b1adcabb2b7d494fb44469769285ad43 P 6435a7db961643ba80a278390d0d97ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6435a7db961643ba80a278390d0d97ec" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 38291 } }
I20240502 20:45:29.036139 23472 raft_consensus.cc:388] T b1adcabb2b7d494fb44469769285ad43 P 6435a7db961643ba80a278390d0d97ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:29.036170 23472 raft_consensus.cc:482] T b1adcabb2b7d494fb44469769285ad43 P 6435a7db961643ba80a278390d0d97ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:29.036203 23472 raft_consensus.cc:3037] T b1adcabb2b7d494fb44469769285ad43 P 6435a7db961643ba80a278390d0d97ec [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:29.036993 23472 raft_consensus.cc:504] T b1adcabb2b7d494fb44469769285ad43 P 6435a7db961643ba80a278390d0d97ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6435a7db961643ba80a278390d0d97ec" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 38291 } }
I20240502 20:45:29.037097 23472 leader_election.cc:302] T b1adcabb2b7d494fb44469769285ad43 P 6435a7db961643ba80a278390d0d97ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6435a7db961643ba80a278390d0d97ec; no voters:
I20240502 20:45:29.037267 23472 leader_election.cc:288] T b1adcabb2b7d494fb44469769285ad43 P 6435a7db961643ba80a278390d0d97ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:29.037338 23487 raft_consensus.cc:2781] T b1adcabb2b7d494fb44469769285ad43 P 6435a7db961643ba80a278390d0d97ec [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:29.037541 23472 ts_tablet_manager.cc:1398] T b1adcabb2b7d494fb44469769285ad43 P 6435a7db961643ba80a278390d0d97ec: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:29.037539 23487 raft_consensus.cc:686] T b1adcabb2b7d494fb44469769285ad43 P 6435a7db961643ba80a278390d0d97ec [term 1 LEADER]: Becoming Leader. State: Replica: 6435a7db961643ba80a278390d0d97ec, State: Running, Role: LEADER
I20240502 20:45:29.037537 23110 catalog_manager.cc:5484] T 5b16516c947042aab223907ea293b3ef P b10d3d589810446888fbbb558b3d1b03 reported cstate change: term changed from 0 to 1, leader changed from <none> to b10d3d589810446888fbbb558b3d1b03 (127.22.104.129). New cstate: current_term: 1 leader_uuid: "b10d3d589810446888fbbb558b3d1b03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b10d3d589810446888fbbb558b3d1b03" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 37329 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:29.037858 23487 consensus_queue.cc:237] T b1adcabb2b7d494fb44469769285ad43 P 6435a7db961643ba80a278390d0d97ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6435a7db961643ba80a278390d0d97ec" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 38291 } }
I20240502 20:45:29.039163 23484 raft_consensus.cc:2781] T 2337eab6e13f401b9f05e5b760ce1d71 P 423887a1c6f84a4f84b53bf6c5052023 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:29.039403 23484 raft_consensus.cc:686] T 2337eab6e13f401b9f05e5b760ce1d71 P 423887a1c6f84a4f84b53bf6c5052023 [term 1 LEADER]: Becoming Leader. State: Replica: 423887a1c6f84a4f84b53bf6c5052023, State: Running, Role: LEADER
I20240502 20:45:29.039633 23484 consensus_queue.cc:237] T 2337eab6e13f401b9f05e5b760ce1d71 P 423887a1c6f84a4f84b53bf6c5052023 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "423887a1c6f84a4f84b53bf6c5052023" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 34059 } }
I20240502 20:45:29.040424 23109 catalog_manager.cc:5484] T b1adcabb2b7d494fb44469769285ad43 P 6435a7db961643ba80a278390d0d97ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 6435a7db961643ba80a278390d0d97ec (127.22.104.130). New cstate: current_term: 1 leader_uuid: "6435a7db961643ba80a278390d0d97ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6435a7db961643ba80a278390d0d97ec" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 38291 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:29.041934 23110 catalog_manager.cc:5484] T 2337eab6e13f401b9f05e5b760ce1d71 P 423887a1c6f84a4f84b53bf6c5052023 reported cstate change: term changed from 0 to 1, leader changed from <none> to 423887a1c6f84a4f84b53bf6c5052023 (127.22.104.131). New cstate: current_term: 1 leader_uuid: "423887a1c6f84a4f84b53bf6c5052023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "423887a1c6f84a4f84b53bf6c5052023" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 34059 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:29.053231 22946 tablet_server.cc:178] TabletServer@127.22.104.129:0 shutting down...
I20240502 20:45:29.057687 22946 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 20:45:29.057932 22946 tablet_replica.cc:330] T 5b16516c947042aab223907ea293b3ef P b10d3d589810446888fbbb558b3d1b03: stopping tablet replica
I20240502 20:45:29.058171 22946 raft_consensus.cc:2229] T 5b16516c947042aab223907ea293b3ef P b10d3d589810446888fbbb558b3d1b03 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:45:29.058297 22946 raft_consensus.cc:2258] T 5b16516c947042aab223907ea293b3ef P b10d3d589810446888fbbb558b3d1b03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:29.059338 22946 tablet_replica.cc:330] T 0d13aad3a9eb4ffab2f7c4480661175c P b10d3d589810446888fbbb558b3d1b03: stopping tablet replica
I20240502 20:45:29.059428 22946 raft_consensus.cc:2229] T 0d13aad3a9eb4ffab2f7c4480661175c P b10d3d589810446888fbbb558b3d1b03 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:45:29.059518 22946 raft_consensus.cc:2258] T 0d13aad3a9eb4ffab2f7c4480661175c P b10d3d589810446888fbbb558b3d1b03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:29.073071 22946 tablet_server.cc:194] TabletServer@127.22.104.129:0 shutdown complete.
I20240502 20:45:29.076835 22946 tablet_server.cc:178] TabletServer@127.22.104.130:0 shutting down...
I20240502 20:45:29.080780 22946 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 20:45:29.081028 22946 tablet_replica.cc:330] T b1adcabb2b7d494fb44469769285ad43 P 6435a7db961643ba80a278390d0d97ec: stopping tablet replica
I20240502 20:45:29.081149 22946 raft_consensus.cc:2229] T b1adcabb2b7d494fb44469769285ad43 P 6435a7db961643ba80a278390d0d97ec [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:45:29.081276 22946 raft_consensus.cc:2258] T b1adcabb2b7d494fb44469769285ad43 P 6435a7db961643ba80a278390d0d97ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:29.094568 22946 tablet_server.cc:194] TabletServer@127.22.104.130:0 shutdown complete.
I20240502 20:45:29.096625 22946 tablet_server.cc:178] TabletServer@127.22.104.131:0 shutting down...
I20240502 20:45:29.099884 22946 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 20:45:29.100176 22946 tablet_replica.cc:330] T 2337eab6e13f401b9f05e5b760ce1d71 P 423887a1c6f84a4f84b53bf6c5052023: stopping tablet replica
I20240502 20:45:29.100296 22946 raft_consensus.cc:2229] T 2337eab6e13f401b9f05e5b760ce1d71 P 423887a1c6f84a4f84b53bf6c5052023 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:45:29.100401 22946 raft_consensus.cc:2258] T 2337eab6e13f401b9f05e5b760ce1d71 P 423887a1c6f84a4f84b53bf6c5052023 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:29.113113 22946 tablet_server.cc:194] TabletServer@127.22.104.131:0 shutdown complete.
I20240502 20:45:29.115288 22946 master.cc:537] Master@127.22.104.190:41025 shutting down...
I20240502 20:45:29.119490 22946 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e5e02cc3547149ae890ef1eb4ebec379 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:29.119642 22946 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e5e02cc3547149ae890ef1eb4ebec379 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:29.119719 22946 tablet_replica.cc:330] T 00000000000000000000000000000000 P e5e02cc3547149ae890ef1eb4ebec379: stopping tablet replica
I20240502 20:45:29.132794 22946 master.cc:552] Master@127.22.104.190:41025 shutdown complete.
I20240502 20:45:29.135797 22946 master.cc:537] Master@127.22.104.189:36603 shutting down...
I20240502 20:45:29.138633 22946 raft_consensus.cc:2229] T 00000000000000000000000000000000 P cd1573c66ef74f41bce4b6d7d30f0576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:29.138835 22946 raft_consensus.cc:2258] T 00000000000000000000000000000000 P cd1573c66ef74f41bce4b6d7d30f0576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:29.138900 22946 tablet_replica.cc:330] T 00000000000000000000000000000000 P cd1573c66ef74f41bce4b6d7d30f0576: stopping tablet replica
I20240502 20:45:29.151391 22946 master.cc:552] Master@127.22.104.189:36603 shutdown complete.
I20240502 20:45:29.153744 22946 master.cc:537] Master@127.22.104.188:38909 shutting down...
I20240502 20:45:29.156868 22946 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:45:29.157152 22946 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:29.157214 22946 tablet_replica.cc:330] T 00000000000000000000000000000000 P bdc2896e260f472b884a650545f0a095: stopping tablet replica
I20240502 20:45:29.170250 22946 master.cc:552] Master@127.22.104.188:38909 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (2638 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20240502 20:45:29.177846 22946 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.22.104.190:32947
I20240502 20:45:29.178167 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:29.180199 23498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:29.180343 23495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:29.180877 23496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:29.181295 22946 server_base.cc:1034] running on GCE node
I20240502 20:45:29.181432 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:45:29.181483 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:45:29.181504 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682729181503 us; error 0 us; skew 500 ppm
I20240502 20:45:29.182281 22946 webserver.cc:460] Webserver started at http://127.22.104.190:42157/ using document root <none> and password file <none>
I20240502 20:45:29.182413 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:29.182453 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:29.182530 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:29.182915 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714682726530369-22946-0/minicluster-data/master-0-root/instance:
uuid: "b73d6348bcd845d3a8febb5a143adea7"
format_stamp: "Formatted at 2024-05-02 20:45:29 on dist-test-slave-rgns"
I20240502 20:45:29.184178 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:29.184962 23503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:29.185184 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:29.185246 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714682726530369-22946-0/minicluster-data/master-0-root
uuid: "b73d6348bcd845d3a8febb5a143adea7"
format_stamp: "Formatted at 2024-05-02 20:45:29 on dist-test-slave-rgns"
I20240502 20:45:29.185310 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714682726530369-22946-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714682726530369-22946-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714682726530369-22946-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:29.189321 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:29.189590 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:29.193135 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.190:32947
I20240502 20:45:29.195565 23555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.190:32947 every 8 connection(s)
I20240502 20:45:29.195991 23556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:29.197680 23556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b73d6348bcd845d3a8febb5a143adea7: Bootstrap starting.
I20240502 20:45:29.198382 23556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b73d6348bcd845d3a8febb5a143adea7: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:29.199313 23556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b73d6348bcd845d3a8febb5a143adea7: No bootstrap required, opened a new log
I20240502 20:45:29.199676 23556 raft_consensus.cc:348] T 00000000000000000000000000000000 P b73d6348bcd845d3a8febb5a143adea7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b73d6348bcd845d3a8febb5a143adea7" member_type: VOTER }
I20240502 20:45:29.199761 23556 raft_consensus.cc:374] T 00000000000000000000000000000000 P b73d6348bcd845d3a8febb5a143adea7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:29.199793 23556 raft_consensus.cc:729] T 00000000000000000000000000000000 P b73d6348bcd845d3a8febb5a143adea7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b73d6348bcd845d3a8febb5a143adea7, State: Initialized, Role: FOLLOWER
I20240502 20:45:29.199918 23556 consensus_queue.cc:260] T 00000000000000000000000000000000 P b73d6348bcd845d3a8febb5a143adea7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b73d6348bcd845d3a8febb5a143adea7" member_type: VOTER }
I20240502 20:45:29.199981 23556 raft_consensus.cc:388] T 00000000000000000000000000000000 P b73d6348bcd845d3a8febb5a143adea7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:29.200011 23556 raft_consensus.cc:482] T 00000000000000000000000000000000 P b73d6348bcd845d3a8febb5a143adea7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:29.200055 23556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b73d6348bcd845d3a8febb5a143adea7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:29.200676 23556 raft_consensus.cc:504] T 00000000000000000000000000000000 P b73d6348bcd845d3a8febb5a143adea7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b73d6348bcd845d3a8febb5a143adea7" member_type: VOTER }
I20240502 20:45:29.200876 23556 leader_election.cc:302] T 00000000000000000000000000000000 P b73d6348bcd845d3a8febb5a143adea7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b73d6348bcd845d3a8febb5a143adea7; no voters:
I20240502 20:45:29.201056 23556 leader_election.cc:288] T 00000000000000000000000000000000 P b73d6348bcd845d3a8febb5a143adea7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:29.201176 23559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b73d6348bcd845d3a8febb5a143adea7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:29.201352 23559 raft_consensus.cc:686] T 00000000000000000000000000000000 P b73d6348bcd845d3a8febb5a143adea7 [term 1 LEADER]: Becoming Leader. State: Replica: b73d6348bcd845d3a8febb5a143adea7, State: Running, Role: LEADER
I20240502 20:45:29.201480 23556 sys_catalog.cc:564] T 00000000000000000000000000000000 P b73d6348bcd845d3a8febb5a143adea7 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 20:45:29.201503 23559 consensus_queue.cc:237] T 00000000000000000000000000000000 P b73d6348bcd845d3a8febb5a143adea7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b73d6348bcd845d3a8febb5a143adea7" member_type: VOTER }
I20240502 20:45:29.201874 23560 sys_catalog.cc:455] T 00000000000000000000000000000000 P b73d6348bcd845d3a8febb5a143adea7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b73d6348bcd845d3a8febb5a143adea7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b73d6348bcd845d3a8febb5a143adea7" member_type: VOTER } }
I20240502 20:45:29.201980 23560 sys_catalog.cc:458] T 00000000000000000000000000000000 P b73d6348bcd845d3a8febb5a143adea7 [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:29.201906 23561 sys_catalog.cc:455] T 00000000000000000000000000000000 P b73d6348bcd845d3a8febb5a143adea7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b73d6348bcd845d3a8febb5a143adea7. Latest consensus state: current_term: 1 leader_uuid: "b73d6348bcd845d3a8febb5a143adea7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b73d6348bcd845d3a8febb5a143adea7" member_type: VOTER } }
I20240502 20:45:29.202112 23561 sys_catalog.cc:458] T 00000000000000000000000000000000 P b73d6348bcd845d3a8febb5a143adea7 [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:29.202538 23564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 20:45:29.203303 23564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 20:45:29.203481 22946 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 20:45:29.205006 23564 catalog_manager.cc:1347] Generated new cluster ID: e93ce39e34ad45edb912aa6595131a21
I20240502 20:45:29.205058 23564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 20:45:29.214999 23564 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 20:45:29.215543 23564 catalog_manager.cc:1504] Loading token signing keys...
I20240502 20:45:29.223644 23564 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b73d6348bcd845d3a8febb5a143adea7: Generated new TSK 0
I20240502 20:45:29.223807 23564 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 20:45:29.235677 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:29.238524 23579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:29.238992 23581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:29.239238 23578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:29.239641 22946 server_base.cc:1034] running on GCE node
I20240502 20:45:29.239791 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:45:29.239828 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:45:29.239847 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682729239847 us; error 0 us; skew 500 ppm
I20240502 20:45:29.240707 22946 webserver.cc:460] Webserver started at http://127.22.104.129:40761/ using document root <none> and password file <none>
I20240502 20:45:29.240851 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:29.240898 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:29.240971 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:29.241333 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714682726530369-22946-0/minicluster-data/ts-0-root/instance:
uuid: "cf520b51778147d08f5fa929459dfd6e"
format_stamp: "Formatted at 2024-05-02 20:45:29 on dist-test-slave-rgns"
I20240502 20:45:29.242764 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:29.243666 23586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:29.243908 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:29.243970 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714682726530369-22946-0/minicluster-data/ts-0-root
uuid: "cf520b51778147d08f5fa929459dfd6e"
format_stamp: "Formatted at 2024-05-02 20:45:29 on dist-test-slave-rgns"
I20240502 20:45:29.244038 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714682726530369-22946-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714682726530369-22946-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714682726530369-22946-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:29.258118 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:29.258489 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:29.258826 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:29.259270 22946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:29.259310 22946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:29.259351 22946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:29.259385 22946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:29.263170 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.129:34807
I20240502 20:45:29.263217 23649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.129:34807 every 8 connection(s)
I20240502 20:45:29.263952 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:29.265918 23654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:29.266208 23655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:29.266659 23657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:29.267158 22946 server_base.cc:1034] running on GCE node
I20240502 20:45:29.267349 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:45:29.267402 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:45:29.267441 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682729267441 us; error 0 us; skew 500 ppm
I20240502 20:45:29.268308 22946 webserver.cc:460] Webserver started at http://127.22.104.130:40377/ using document root <none> and password file <none>
I20240502 20:45:29.268473 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:29.268473 23650 heartbeater.cc:347] Connected to a master server at 127.22.104.190:32947
I20240502 20:45:29.268548 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:29.268586 23650 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:29.268622 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:29.268817 23650 heartbeater.cc:511] Master 127.22.104.190:32947 requested a full tablet report, sending...
I20240502 20:45:29.269073 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714682726530369-22946-0/minicluster-data/ts-1-root/instance:
uuid: "ff8bccc72e754c8cba77ead752f3fc4d"
format_stamp: "Formatted at 2024-05-02 20:45:29 on dist-test-slave-rgns"
I20240502 20:45:29.269454 23520 ts_manager.cc:194] Registered new tserver with Master: cf520b51778147d08f5fa929459dfd6e (127.22.104.129:34807)
I20240502 20:45:29.270138 23520 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48950
I20240502 20:45:29.270578 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 20:45:29.271490 23663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:29.271728 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:29.271790 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714682726530369-22946-0/minicluster-data/ts-1-root
uuid: "ff8bccc72e754c8cba77ead752f3fc4d"
format_stamp: "Formatted at 2024-05-02 20:45:29 on dist-test-slave-rgns"
I20240502 20:45:29.271858 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714682726530369-22946-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714682726530369-22946-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714682726530369-22946-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:29.298839 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:29.299232 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:29.299517 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:29.299961 22946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:29.299999 22946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:29.300052 22946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:29.300081 22946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:29.304013 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.130:43443
I20240502 20:45:29.304087 23726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.130:43443 every 8 connection(s)
I20240502 20:45:29.304733 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:29.306905 23731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:29.307360 23733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:29.307760 22946 server_base.cc:1034] running on GCE node
I20240502 20:45:29.308758 23727 heartbeater.cc:347] Connected to a master server at 127.22.104.190:32947
W20240502 20:45:29.308835 23730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:29.308861 23727 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:29.309064 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240502 20:45:29.309095 23727 heartbeater.cc:511] Master 127.22.104.190:32947 requested a full tablet report, sending...
W20240502 20:45:29.309113 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:45:29.309137 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682729309136 us; error 0 us; skew 500 ppm
I20240502 20:45:29.309720 23520 ts_manager.cc:194] Registered new tserver with Master: ff8bccc72e754c8cba77ead752f3fc4d (127.22.104.130:43443)
I20240502 20:45:29.310029 22946 webserver.cc:460] Webserver started at http://127.22.104.131:37465/ using document root <none> and password file <none>
I20240502 20:45:29.310168 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:29.310213 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:29.310285 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:29.310365 23520 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48962
I20240502 20:45:29.310760 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714682726530369-22946-0/minicluster-data/ts-2-root/instance:
uuid: "39e2d0baa860401bbf588f9f9e03fef2"
format_stamp: "Formatted at 2024-05-02 20:45:29 on dist-test-slave-rgns"
I20240502 20:45:29.312157 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:29.312985 23738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:29.313235 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:29.313299 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714682726530369-22946-0/minicluster-data/ts-2-root
uuid: "39e2d0baa860401bbf588f9f9e03fef2"
format_stamp: "Formatted at 2024-05-02 20:45:29 on dist-test-slave-rgns"
I20240502 20:45:29.313364 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714682726530369-22946-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714682726530369-22946-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714682726530369-22946-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:29.319558 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:29.319869 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:29.320134 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:29.320529 22946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:29.320566 22946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:29.320602 22946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:29.320636 22946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:29.324821 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.131:35163
I20240502 20:45:29.324882 23801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.131:35163 every 8 connection(s)
I20240502 20:45:29.328773 23802 heartbeater.cc:347] Connected to a master server at 127.22.104.190:32947
I20240502 20:45:29.328853 23802 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:29.329010 23802 heartbeater.cc:511] Master 127.22.104.190:32947 requested a full tablet report, sending...
I20240502 20:45:29.329543 23520 ts_manager.cc:194] Registered new tserver with Master: 39e2d0baa860401bbf588f9f9e03fef2 (127.22.104.131:35163)
I20240502 20:45:29.329689 22946 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004499506s
I20240502 20:45:29.330032 23520 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48976
I20240502 20:45:30.271092 23650 heartbeater.cc:503] Master 127.22.104.190:32947 was elected leader, sending a full tablet report...
I20240502 20:45:30.311301 23727 heartbeater.cc:503] Master 127.22.104.190:32947 was elected leader, sending a full tablet report...
I20240502 20:45:30.330839 23802 heartbeater.cc:503] Master 127.22.104.190:32947 was elected leader, sending a full tablet report...
I20240502 20:45:30.355736 22946 test_util.cc:268] Using random seed: -1358193644
I20240502 20:45:30.363794 23520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48992:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 20:45:30.364403 23520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 20:45:30.376892 23764 tablet_service.cc:1467] Processing CreateTablet for tablet 74948efa3e364ee596ea1050cddaaae2 (DEFAULT_TABLE table=test-workload [id=4c3b89d9641344a8b05408d876f21066]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 20:45:30.377163 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 13f3f643f725403eb35362f87f389145 (DEFAULT_TABLE table=test-workload [id=4c3b89d9641344a8b05408d876f21066]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 20:45:30.377163 23766 tablet_service.cc:1467] Processing CreateTablet for tablet a37266ada21d4a17b45bdee1225c7e1a (DEFAULT_TABLE table=test-workload [id=4c3b89d9641344a8b05408d876f21066]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 20:45:30.377288 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74948efa3e364ee596ea1050cddaaae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:30.377650 23760 tablet_service.cc:1467] Processing CreateTablet for tablet 1ee72890ba03492b9eebab393765739f (DEFAULT_TABLE table=test-workload [id=4c3b89d9641344a8b05408d876f21066]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 20:45:30.377662 23614 tablet_service.cc:1467] Processing CreateTablet for tablet a37266ada21d4a17b45bdee1225c7e1a (DEFAULT_TABLE table=test-workload [id=4c3b89d9641344a8b05408d876f21066]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 20:45:30.377924 23614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a37266ada21d4a17b45bdee1225c7e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:30.377980 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ee72890ba03492b9eebab393765739f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:30.378077 23761 tablet_service.cc:1467] Processing CreateTablet for tablet 6f7bab52adb145df84de04cb1b7ea315 (DEFAULT_TABLE table=test-workload [id=4c3b89d9641344a8b05408d876f21066]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 20:45:30.378289 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f7bab52adb145df84de04cb1b7ea315. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:30.380268 23613 tablet_service.cc:1467] Processing CreateTablet for tablet 49f3469a24a147c2a9b0f2634e2112e7 (DEFAULT_TABLE table=test-workload [id=4c3b89d9641344a8b05408d876f21066]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 20:45:30.380447 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 0fc3ca8a4b274046b1c1842318c8db21 (DEFAULT_TABLE table=test-workload [id=4c3b89d9641344a8b05408d876f21066]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 20:45:30.380548 23607 tablet_service.cc:1467] Processing CreateTablet for tablet 13f3f643f725403eb35362f87f389145 (DEFAULT_TABLE table=test-workload [id=4c3b89d9641344a8b05408d876f21066]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 20:45:30.380559 23613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49f3469a24a147c2a9b0f2634e2112e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:30.380837 23762 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa05319be264a8dacfba3fbb757629b (DEFAULT_TABLE table=test-workload [id=4c3b89d9641344a8b05408d876f21066]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 20:45:30.380982 23609 tablet_service.cc:1467] Processing CreateTablet for tablet 6f7bab52adb145df84de04cb1b7ea315 (DEFAULT_TABLE table=test-workload [id=4c3b89d9641344a8b05408d876f21066]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 20:45:30.381242 23609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f7bab52adb145df84de04cb1b7ea315. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:30.381192 23612 tablet_service.cc:1467] Processing CreateTablet for tablet 74948efa3e364ee596ea1050cddaaae2 (DEFAULT_TABLE table=test-workload [id=4c3b89d9641344a8b05408d876f21066]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 20:45:30.381392 23612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74948efa3e364ee596ea1050cddaaae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:30.380268 23765 tablet_service.cc:1467] Processing CreateTablet for tablet 49f3469a24a147c2a9b0f2634e2112e7 (DEFAULT_TABLE table=test-workload [id=4c3b89d9641344a8b05408d876f21066]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 20:45:30.380982 23608 tablet_service.cc:1467] Processing CreateTablet for tablet 1ee72890ba03492b9eebab393765739f (DEFAULT_TABLE table=test-workload [id=4c3b89d9641344a8b05408d876f21066]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 20:45:30.381829 23608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ee72890ba03492b9eebab393765739f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:30.381878 23823 tablet_bootstrap.cc:492] T a37266ada21d4a17b45bdee1225c7e1a P cf520b51778147d08f5fa929459dfd6e: Bootstrap starting.
I20240502 20:45:30.380292 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a37266ada21d4a17b45bdee1225c7e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:30.383170 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13f3f643f725403eb35362f87f389145. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:30.383354 23611 tablet_service.cc:1467] Processing CreateTablet for tablet 0fc3ca8a4b274046b1c1842318c8db21 (DEFAULT_TABLE table=test-workload [id=4c3b89d9641344a8b05408d876f21066]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 20:45:30.383564 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49f3469a24a147c2a9b0f2634e2112e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:30.383594 23611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc3ca8a4b274046b1c1842318c8db21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:30.384003 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc3ca8a4b274046b1c1842318c8db21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:30.385483 23607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13f3f643f725403eb35362f87f389145. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:30.385601 23610 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa05319be264a8dacfba3fbb757629b (DEFAULT_TABLE table=test-workload [id=4c3b89d9641344a8b05408d876f21066]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 20:45:30.385814 23610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa05319be264a8dacfba3fbb757629b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:30.385851 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa05319be264a8dacfba3fbb757629b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:30.388046 23823 tablet_bootstrap.cc:654] T a37266ada21d4a17b45bdee1225c7e1a P cf520b51778147d08f5fa929459dfd6e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.389046 23691 tablet_service.cc:1467] Processing CreateTablet for tablet a37266ada21d4a17b45bdee1225c7e1a (DEFAULT_TABLE table=test-workload [id=4c3b89d9641344a8b05408d876f21066]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 20:45:30.388568 23822 tablet_bootstrap.cc:492] T 74948efa3e364ee596ea1050cddaaae2 P 39e2d0baa860401bbf588f9f9e03fef2: Bootstrap starting.
I20240502 20:45:30.389292 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a37266ada21d4a17b45bdee1225c7e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:30.389634 23684 tablet_service.cc:1467] Processing CreateTablet for tablet 13f3f643f725403eb35362f87f389145 (DEFAULT_TABLE table=test-workload [id=4c3b89d9641344a8b05408d876f21066]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 20:45:30.389788 23823 tablet_bootstrap.cc:492] T a37266ada21d4a17b45bdee1225c7e1a P cf520b51778147d08f5fa929459dfd6e: No bootstrap required, opened a new log
I20240502 20:45:30.389856 23684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13f3f643f725403eb35362f87f389145. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:30.389868 23823 ts_tablet_manager.cc:1367] T a37266ada21d4a17b45bdee1225c7e1a P cf520b51778147d08f5fa929459dfd6e: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 20:45:30.390089 23822 tablet_bootstrap.cc:654] T 74948efa3e364ee596ea1050cddaaae2 P 39e2d0baa860401bbf588f9f9e03fef2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.390340 23823 raft_consensus.cc:348] T a37266ada21d4a17b45bdee1225c7e1a P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } } peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } }
I20240502 20:45:30.390453 23823 raft_consensus.cc:374] T a37266ada21d4a17b45bdee1225c7e1a P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.390484 23823 raft_consensus.cc:729] T a37266ada21d4a17b45bdee1225c7e1a P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf520b51778147d08f5fa929459dfd6e, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.390602 23823 consensus_queue.cc:260] T a37266ada21d4a17b45bdee1225c7e1a P cf520b51778147d08f5fa929459dfd6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } } peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } }
I20240502 20:45:30.390930 23690 tablet_service.cc:1467] Processing CreateTablet for tablet 49f3469a24a147c2a9b0f2634e2112e7 (DEFAULT_TABLE table=test-workload [id=4c3b89d9641344a8b05408d876f21066]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 20:45:30.390985 23823 ts_tablet_manager.cc:1398] T a37266ada21d4a17b45bdee1225c7e1a P cf520b51778147d08f5fa929459dfd6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:30.391199 23823 tablet_bootstrap.cc:492] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e: Bootstrap starting.
I20240502 20:45:30.391911 23689 tablet_service.cc:1467] Processing CreateTablet for tablet 74948efa3e364ee596ea1050cddaaae2 (DEFAULT_TABLE table=test-workload [id=4c3b89d9641344a8b05408d876f21066]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 20:45:30.391994 23690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49f3469a24a147c2a9b0f2634e2112e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:30.392467 23827 tablet_bootstrap.cc:492] T a37266ada21d4a17b45bdee1225c7e1a P ff8bccc72e754c8cba77ead752f3fc4d: Bootstrap starting.
I20240502 20:45:30.393463 23827 tablet_bootstrap.cc:654] T a37266ada21d4a17b45bdee1225c7e1a P ff8bccc72e754c8cba77ead752f3fc4d: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.393700 23685 tablet_service.cc:1467] Processing CreateTablet for tablet 1ee72890ba03492b9eebab393765739f (DEFAULT_TABLE table=test-workload [id=4c3b89d9641344a8b05408d876f21066]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 20:45:30.393739 23687 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa05319be264a8dacfba3fbb757629b (DEFAULT_TABLE table=test-workload [id=4c3b89d9641344a8b05408d876f21066]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 20:45:30.393888 23688 tablet_service.cc:1467] Processing CreateTablet for tablet 0fc3ca8a4b274046b1c1842318c8db21 (DEFAULT_TABLE table=test-workload [id=4c3b89d9641344a8b05408d876f21066]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 20:45:30.394134 23688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc3ca8a4b274046b1c1842318c8db21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:30.394243 23687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa05319be264a8dacfba3fbb757629b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:30.394299 23686 tablet_service.cc:1467] Processing CreateTablet for tablet 6f7bab52adb145df84de04cb1b7ea315 (DEFAULT_TABLE table=test-workload [id=4c3b89d9641344a8b05408d876f21066]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 20:45:30.394562 23686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f7bab52adb145df84de04cb1b7ea315. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:30.394847 23827 tablet_bootstrap.cc:492] T a37266ada21d4a17b45bdee1225c7e1a P ff8bccc72e754c8cba77ead752f3fc4d: No bootstrap required, opened a new log
I20240502 20:45:30.394925 23827 ts_tablet_manager.cc:1367] T a37266ada21d4a17b45bdee1225c7e1a P ff8bccc72e754c8cba77ead752f3fc4d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:45:30.395337 23827 raft_consensus.cc:348] T a37266ada21d4a17b45bdee1225c7e1a P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } } peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } }
I20240502 20:45:30.395437 23827 raft_consensus.cc:374] T a37266ada21d4a17b45bdee1225c7e1a P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.395468 23827 raft_consensus.cc:729] T a37266ada21d4a17b45bdee1225c7e1a P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff8bccc72e754c8cba77ead752f3fc4d, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.395581 23827 consensus_queue.cc:260] T a37266ada21d4a17b45bdee1225c7e1a P ff8bccc72e754c8cba77ead752f3fc4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } } peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } }
I20240502 20:45:30.395932 23827 ts_tablet_manager.cc:1398] T a37266ada21d4a17b45bdee1225c7e1a P ff8bccc72e754c8cba77ead752f3fc4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:30.396217 23689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74948efa3e364ee596ea1050cddaaae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:30.396359 23827 tablet_bootstrap.cc:492] T 13f3f643f725403eb35362f87f389145 P ff8bccc72e754c8cba77ead752f3fc4d: Bootstrap starting.
I20240502 20:45:30.397192 23822 tablet_bootstrap.cc:492] T 74948efa3e364ee596ea1050cddaaae2 P 39e2d0baa860401bbf588f9f9e03fef2: No bootstrap required, opened a new log
I20240502 20:45:30.397224 23827 tablet_bootstrap.cc:654] T 13f3f643f725403eb35362f87f389145 P ff8bccc72e754c8cba77ead752f3fc4d: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.397269 23822 ts_tablet_manager.cc:1367] T 74948efa3e364ee596ea1050cddaaae2 P 39e2d0baa860401bbf588f9f9e03fef2: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 20:45:30.397661 23822 raft_consensus.cc:348] T 74948efa3e364ee596ea1050cddaaae2 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.397755 23685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ee72890ba03492b9eebab393765739f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:30.397756 23822 raft_consensus.cc:374] T 74948efa3e364ee596ea1050cddaaae2 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.397876 23822 raft_consensus.cc:729] T 74948efa3e364ee596ea1050cddaaae2 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39e2d0baa860401bbf588f9f9e03fef2, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.398140 23822 consensus_queue.cc:260] T 74948efa3e364ee596ea1050cddaaae2 P 39e2d0baa860401bbf588f9f9e03fef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.398547 23822 ts_tablet_manager.cc:1398] T 74948efa3e364ee596ea1050cddaaae2 P 39e2d0baa860401bbf588f9f9e03fef2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:30.398720 23823 tablet_bootstrap.cc:654] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.399169 23822 tablet_bootstrap.cc:492] T 1ee72890ba03492b9eebab393765739f P 39e2d0baa860401bbf588f9f9e03fef2: Bootstrap starting.
I20240502 20:45:30.399727 23827 tablet_bootstrap.cc:492] T 13f3f643f725403eb35362f87f389145 P ff8bccc72e754c8cba77ead752f3fc4d: No bootstrap required, opened a new log
I20240502 20:45:30.399819 23827 ts_tablet_manager.cc:1367] T 13f3f643f725403eb35362f87f389145 P ff8bccc72e754c8cba77ead752f3fc4d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:45:30.400208 23827 raft_consensus.cc:348] T 13f3f643f725403eb35362f87f389145 P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.400315 23827 raft_consensus.cc:374] T 13f3f643f725403eb35362f87f389145 P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.400354 23827 raft_consensus.cc:729] T 13f3f643f725403eb35362f87f389145 P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff8bccc72e754c8cba77ead752f3fc4d, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.400481 23827 consensus_queue.cc:260] T 13f3f643f725403eb35362f87f389145 P ff8bccc72e754c8cba77ead752f3fc4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.400729 23827 ts_tablet_manager.cc:1398] T 13f3f643f725403eb35362f87f389145 P ff8bccc72e754c8cba77ead752f3fc4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:30.400878 23827 tablet_bootstrap.cc:492] T 49f3469a24a147c2a9b0f2634e2112e7 P ff8bccc72e754c8cba77ead752f3fc4d: Bootstrap starting.
I20240502 20:45:30.400996 23823 tablet_bootstrap.cc:492] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e: No bootstrap required, opened a new log
I20240502 20:45:30.401104 23823 ts_tablet_manager.cc:1367] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 20:45:30.401472 23829 raft_consensus.cc:482] T a37266ada21d4a17b45bdee1225c7e1a P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:30.401543 23829 raft_consensus.cc:504] T a37266ada21d4a17b45bdee1225c7e1a P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } } peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } }
I20240502 20:45:30.401739 23827 tablet_bootstrap.cc:654] T 49f3469a24a147c2a9b0f2634e2112e7 P ff8bccc72e754c8cba77ead752f3fc4d: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.401551 23823 raft_consensus.cc:348] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.401855 23823 raft_consensus.cc:374] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.401932 23823 raft_consensus.cc:729] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf520b51778147d08f5fa929459dfd6e, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.402031 23829 leader_election.cc:288] T a37266ada21d4a17b45bdee1225c7e1a P ff8bccc72e754c8cba77ead752f3fc4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39e2d0baa860401bbf588f9f9e03fef2 (127.22.104.131:35163), cf520b51778147d08f5fa929459dfd6e (127.22.104.129:34807)
I20240502 20:45:30.402088 23822 tablet_bootstrap.cc:654] T 1ee72890ba03492b9eebab393765739f P 39e2d0baa860401bbf588f9f9e03fef2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.402123 23823 consensus_queue.cc:260] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.402953 23823 ts_tablet_manager.cc:1398] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:30.403163 23823 tablet_bootstrap.cc:492] T 74948efa3e364ee596ea1050cddaaae2 P cf520b51778147d08f5fa929459dfd6e: Bootstrap starting.
I20240502 20:45:30.403228 23827 tablet_bootstrap.cc:492] T 49f3469a24a147c2a9b0f2634e2112e7 P ff8bccc72e754c8cba77ead752f3fc4d: No bootstrap required, opened a new log
I20240502 20:45:30.403352 23827 ts_tablet_manager.cc:1367] T 49f3469a24a147c2a9b0f2634e2112e7 P ff8bccc72e754c8cba77ead752f3fc4d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:45:30.403784 23827 raft_consensus.cc:348] T 49f3469a24a147c2a9b0f2634e2112e7 P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.403909 23827 raft_consensus.cc:374] T 49f3469a24a147c2a9b0f2634e2112e7 P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.403980 23827 raft_consensus.cc:729] T 49f3469a24a147c2a9b0f2634e2112e7 P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff8bccc72e754c8cba77ead752f3fc4d, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.404105 23827 consensus_queue.cc:260] T 49f3469a24a147c2a9b0f2634e2112e7 P ff8bccc72e754c8cba77ead752f3fc4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.404361 23827 ts_tablet_manager.cc:1398] T 49f3469a24a147c2a9b0f2634e2112e7 P ff8bccc72e754c8cba77ead752f3fc4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:30.404544 23827 tablet_bootstrap.cc:492] T 0fc3ca8a4b274046b1c1842318c8db21 P ff8bccc72e754c8cba77ead752f3fc4d: Bootstrap starting.
I20240502 20:45:30.405442 23827 tablet_bootstrap.cc:654] T 0fc3ca8a4b274046b1c1842318c8db21 P ff8bccc72e754c8cba77ead752f3fc4d: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.406760 23823 tablet_bootstrap.cc:654] T 74948efa3e364ee596ea1050cddaaae2 P cf520b51778147d08f5fa929459dfd6e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.407442 23776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37266ada21d4a17b45bdee1225c7e1a" candidate_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39e2d0baa860401bbf588f9f9e03fef2" is_pre_election: true
I20240502 20:45:30.407805 23827 tablet_bootstrap.cc:492] T 0fc3ca8a4b274046b1c1842318c8db21 P ff8bccc72e754c8cba77ead752f3fc4d: No bootstrap required, opened a new log
I20240502 20:45:30.407922 23827 ts_tablet_manager.cc:1367] T 0fc3ca8a4b274046b1c1842318c8db21 P ff8bccc72e754c8cba77ead752f3fc4d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
W20240502 20:45:30.407982 23667 leader_election.cc:341] T a37266ada21d4a17b45bdee1225c7e1a P ff8bccc72e754c8cba77ead752f3fc4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39e2d0baa860401bbf588f9f9e03fef2 (127.22.104.131:35163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 20:45:30.408337 23822 tablet_bootstrap.cc:492] T 1ee72890ba03492b9eebab393765739f P 39e2d0baa860401bbf588f9f9e03fef2: No bootstrap required, opened a new log
I20240502 20:45:30.408432 23823 tablet_bootstrap.cc:492] T 74948efa3e364ee596ea1050cddaaae2 P cf520b51778147d08f5fa929459dfd6e: No bootstrap required, opened a new log
I20240502 20:45:30.408352 23827 raft_consensus.cc:348] T 0fc3ca8a4b274046b1c1842318c8db21 P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.408470 23822 ts_tablet_manager.cc:1367] T 1ee72890ba03492b9eebab393765739f P 39e2d0baa860401bbf588f9f9e03fef2: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 20:45:30.408473 23827 raft_consensus.cc:374] T 0fc3ca8a4b274046b1c1842318c8db21 P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.408500 23823 ts_tablet_manager.cc:1367] T 74948efa3e364ee596ea1050cddaaae2 P cf520b51778147d08f5fa929459dfd6e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 20:45:30.408542 23827 raft_consensus.cc:729] T 0fc3ca8a4b274046b1c1842318c8db21 P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff8bccc72e754c8cba77ead752f3fc4d, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.408705 23827 consensus_queue.cc:260] T 0fc3ca8a4b274046b1c1842318c8db21 P ff8bccc72e754c8cba77ead752f3fc4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.408814 23829 raft_consensus.cc:482] T 0fc3ca8a4b274046b1c1842318c8db21 P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:30.408841 23822 raft_consensus.cc:348] T 1ee72890ba03492b9eebab393765739f P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.408885 23829 raft_consensus.cc:504] T 0fc3ca8a4b274046b1c1842318c8db21 P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.408933 23822 raft_consensus.cc:374] T 1ee72890ba03492b9eebab393765739f P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.408962 23822 raft_consensus.cc:729] T 1ee72890ba03492b9eebab393765739f P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39e2d0baa860401bbf588f9f9e03fef2, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.409051 23827 ts_tablet_manager.cc:1398] T 0fc3ca8a4b274046b1c1842318c8db21 P ff8bccc72e754c8cba77ead752f3fc4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:30.408325 23624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37266ada21d4a17b45bdee1225c7e1a" candidate_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf520b51778147d08f5fa929459dfd6e" is_pre_election: true
I20240502 20:45:30.409224 23829 leader_election.cc:288] T 0fc3ca8a4b274046b1c1842318c8db21 P ff8bccc72e754c8cba77ead752f3fc4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf520b51778147d08f5fa929459dfd6e (127.22.104.129:34807), 39e2d0baa860401bbf588f9f9e03fef2 (127.22.104.131:35163)
I20240502 20:45:30.409068 23822 consensus_queue.cc:260] T 1ee72890ba03492b9eebab393765739f P 39e2d0baa860401bbf588f9f9e03fef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.409209 23827 tablet_bootstrap.cc:492] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d: Bootstrap starting.
I20240502 20:45:30.409538 23822 ts_tablet_manager.cc:1398] T 1ee72890ba03492b9eebab393765739f P 39e2d0baa860401bbf588f9f9e03fef2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:30.409605 23623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc3ca8a4b274046b1c1842318c8db21" candidate_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf520b51778147d08f5fa929459dfd6e" is_pre_election: true
W20240502 20:45:30.410037 23667 leader_election.cc:341] T 0fc3ca8a4b274046b1c1842318c8db21 P ff8bccc72e754c8cba77ead752f3fc4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cf520b51778147d08f5fa929459dfd6e (127.22.104.129:34807): Illegal state: must be running to vote when last-logged opid is not known
I20240502 20:45:30.410221 23827 tablet_bootstrap.cc:654] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.409206 23624 raft_consensus.cc:2454] T a37266ada21d4a17b45bdee1225c7e1a P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff8bccc72e754c8cba77ead752f3fc4d in term 0.
I20240502 20:45:30.410550 23667 leader_election.cc:302] T a37266ada21d4a17b45bdee1225c7e1a P ff8bccc72e754c8cba77ead752f3fc4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf520b51778147d08f5fa929459dfd6e, ff8bccc72e754c8cba77ead752f3fc4d; no voters: 39e2d0baa860401bbf588f9f9e03fef2
I20240502 20:45:30.410579 23823 raft_consensus.cc:348] T 74948efa3e364ee596ea1050cddaaae2 P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.410663 23823 raft_consensus.cc:374] T 74948efa3e364ee596ea1050cddaaae2 P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.410717 23823 raft_consensus.cc:729] T 74948efa3e364ee596ea1050cddaaae2 P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf520b51778147d08f5fa929459dfd6e, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.410828 23823 consensus_queue.cc:260] T 74948efa3e364ee596ea1050cddaaae2 P cf520b51778147d08f5fa929459dfd6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.410869 23776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc3ca8a4b274046b1c1842318c8db21" candidate_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39e2d0baa860401bbf588f9f9e03fef2" is_pre_election: true
I20240502 20:45:30.411152 23823 ts_tablet_manager.cc:1398] T 74948efa3e364ee596ea1050cddaaae2 P cf520b51778147d08f5fa929459dfd6e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
W20240502 20:45:30.411217 23667 leader_election.cc:341] T 0fc3ca8a4b274046b1c1842318c8db21 P ff8bccc72e754c8cba77ead752f3fc4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39e2d0baa860401bbf588f9f9e03fef2 (127.22.104.131:35163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 20:45:30.411298 23667 leader_election.cc:302] T 0fc3ca8a4b274046b1c1842318c8db21 P ff8bccc72e754c8cba77ead752f3fc4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff8bccc72e754c8cba77ead752f3fc4d; no voters: 39e2d0baa860401bbf588f9f9e03fef2, cf520b51778147d08f5fa929459dfd6e
I20240502 20:45:30.411473 23823 tablet_bootstrap.cc:492] T 0fc3ca8a4b274046b1c1842318c8db21 P cf520b51778147d08f5fa929459dfd6e: Bootstrap starting.
I20240502 20:45:30.411496 23829 raft_consensus.cc:2781] T a37266ada21d4a17b45bdee1225c7e1a P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:30.409698 23822 tablet_bootstrap.cc:492] T 6f7bab52adb145df84de04cb1b7ea315 P 39e2d0baa860401bbf588f9f9e03fef2: Bootstrap starting.
I20240502 20:45:30.411587 23829 raft_consensus.cc:482] T a37266ada21d4a17b45bdee1225c7e1a P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:30.411670 23829 raft_consensus.cc:3037] T a37266ada21d4a17b45bdee1225c7e1a P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.412441 23822 tablet_bootstrap.cc:654] T 6f7bab52adb145df84de04cb1b7ea315 P 39e2d0baa860401bbf588f9f9e03fef2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.412608 23829 raft_consensus.cc:504] T a37266ada21d4a17b45bdee1225c7e1a P ff8bccc72e754c8cba77ead752f3fc4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } } peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } }
I20240502 20:45:30.412889 23829 leader_election.cc:288] T a37266ada21d4a17b45bdee1225c7e1a P ff8bccc72e754c8cba77ead752f3fc4d [CANDIDATE]: Term 1 election: Requested vote from peers 39e2d0baa860401bbf588f9f9e03fef2 (127.22.104.131:35163), cf520b51778147d08f5fa929459dfd6e (127.22.104.129:34807)
I20240502 20:45:30.413553 23624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37266ada21d4a17b45bdee1225c7e1a" candidate_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf520b51778147d08f5fa929459dfd6e"
I20240502 20:45:30.413650 23624 raft_consensus.cc:3037] T a37266ada21d4a17b45bdee1225c7e1a P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.413831 23823 tablet_bootstrap.cc:654] T 0fc3ca8a4b274046b1c1842318c8db21 P cf520b51778147d08f5fa929459dfd6e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.414393 23624 raft_consensus.cc:2454] T a37266ada21d4a17b45bdee1225c7e1a P cf520b51778147d08f5fa929459dfd6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff8bccc72e754c8cba77ead752f3fc4d in term 1.
I20240502 20:45:30.411585 23839 raft_consensus.cc:2726] T 0fc3ca8a4b274046b1c1842318c8db21 P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 20:45:30.414876 23822 tablet_bootstrap.cc:492] T 6f7bab52adb145df84de04cb1b7ea315 P 39e2d0baa860401bbf588f9f9e03fef2: No bootstrap required, opened a new log
I20240502 20:45:30.415020 23667 leader_election.cc:302] T a37266ada21d4a17b45bdee1225c7e1a P ff8bccc72e754c8cba77ead752f3fc4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf520b51778147d08f5fa929459dfd6e, ff8bccc72e754c8cba77ead752f3fc4d; no voters:
I20240502 20:45:30.415050 23823 tablet_bootstrap.cc:492] T 0fc3ca8a4b274046b1c1842318c8db21 P cf520b51778147d08f5fa929459dfd6e: No bootstrap required, opened a new log
I20240502 20:45:30.415107 23822 ts_tablet_manager.cc:1367] T 6f7bab52adb145df84de04cb1b7ea315 P 39e2d0baa860401bbf588f9f9e03fef2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 20:45:30.415134 23823 ts_tablet_manager.cc:1367] T 0fc3ca8a4b274046b1c1842318c8db21 P cf520b51778147d08f5fa929459dfd6e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:45:30.415500 23823 raft_consensus.cc:348] T 0fc3ca8a4b274046b1c1842318c8db21 P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.414880 23776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37266ada21d4a17b45bdee1225c7e1a" candidate_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39e2d0baa860401bbf588f9f9e03fef2"
I20240502 20:45:30.415601 23823 raft_consensus.cc:374] T 0fc3ca8a4b274046b1c1842318c8db21 P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.415642 23827 tablet_bootstrap.cc:492] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d: No bootstrap required, opened a new log
I20240502 20:45:30.415642 23823 raft_consensus.cc:729] T 0fc3ca8a4b274046b1c1842318c8db21 P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf520b51778147d08f5fa929459dfd6e, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.415733 23827 ts_tablet_manager.cc:1367] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240502 20:45:30.415609 23822 raft_consensus.cc:348] T 6f7bab52adb145df84de04cb1b7ea315 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.415827 23823 consensus_queue.cc:260] T 0fc3ca8a4b274046b1c1842318c8db21 P cf520b51778147d08f5fa929459dfd6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.415827 23822 raft_consensus.cc:374] T 6f7bab52adb145df84de04cb1b7ea315 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240502 20:45:30.415918 23667 leader_election.cc:341] T a37266ada21d4a17b45bdee1225c7e1a P ff8bccc72e754c8cba77ead752f3fc4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 39e2d0baa860401bbf588f9f9e03fef2 (127.22.104.131:35163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 20:45:30.415971 23822 raft_consensus.cc:729] T 6f7bab52adb145df84de04cb1b7ea315 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39e2d0baa860401bbf588f9f9e03fef2, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.416070 23822 consensus_queue.cc:260] T 6f7bab52adb145df84de04cb1b7ea315 P 39e2d0baa860401bbf588f9f9e03fef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.416118 23823 ts_tablet_manager.cc:1398] T 0fc3ca8a4b274046b1c1842318c8db21 P cf520b51778147d08f5fa929459dfd6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:30.416136 23827 raft_consensus.cc:348] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.416222 23827 raft_consensus.cc:374] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.416252 23827 raft_consensus.cc:729] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff8bccc72e754c8cba77ead752f3fc4d, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.416265 23823 tablet_bootstrap.cc:492] T 13f3f643f725403eb35362f87f389145 P cf520b51778147d08f5fa929459dfd6e: Bootstrap starting.
I20240502 20:45:30.416286 23822 ts_tablet_manager.cc:1398] T 6f7bab52adb145df84de04cb1b7ea315 P 39e2d0baa860401bbf588f9f9e03fef2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:30.416357 23827 consensus_queue.cc:260] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.416420 23822 tablet_bootstrap.cc:492] T a37266ada21d4a17b45bdee1225c7e1a P 39e2d0baa860401bbf588f9f9e03fef2: Bootstrap starting.
I20240502 20:45:30.416739 23827 ts_tablet_manager.cc:1398] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:45:30.417208 23823 tablet_bootstrap.cc:654] T 13f3f643f725403eb35362f87f389145 P cf520b51778147d08f5fa929459dfd6e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.417255 23827 tablet_bootstrap.cc:492] T 6f7bab52adb145df84de04cb1b7ea315 P ff8bccc72e754c8cba77ead752f3fc4d: Bootstrap starting.
I20240502 20:45:30.417208 23822 tablet_bootstrap.cc:654] T a37266ada21d4a17b45bdee1225c7e1a P 39e2d0baa860401bbf588f9f9e03fef2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.418123 23827 tablet_bootstrap.cc:654] T 6f7bab52adb145df84de04cb1b7ea315 P ff8bccc72e754c8cba77ead752f3fc4d: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.418208 23826 raft_consensus.cc:482] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:30.418279 23826 raft_consensus.cc:504] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.418505 23829 raft_consensus.cc:2781] T a37266ada21d4a17b45bdee1225c7e1a P ff8bccc72e754c8cba77ead752f3fc4d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:30.418674 23829 raft_consensus.cc:686] T a37266ada21d4a17b45bdee1225c7e1a P ff8bccc72e754c8cba77ead752f3fc4d [term 1 LEADER]: Becoming Leader. State: Replica: ff8bccc72e754c8cba77ead752f3fc4d, State: Running, Role: LEADER
I20240502 20:45:30.418745 23826 leader_election.cc:288] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff8bccc72e754c8cba77ead752f3fc4d (127.22.104.130:43443), 39e2d0baa860401bbf588f9f9e03fef2 (127.22.104.131:35163)
I20240502 20:45:30.418812 23829 consensus_queue.cc:237] T a37266ada21d4a17b45bdee1225c7e1a P ff8bccc72e754c8cba77ead752f3fc4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } } peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } }
I20240502 20:45:30.420311 23822 tablet_bootstrap.cc:492] T a37266ada21d4a17b45bdee1225c7e1a P 39e2d0baa860401bbf588f9f9e03fef2: No bootstrap required, opened a new log
I20240502 20:45:30.420398 23822 ts_tablet_manager.cc:1367] T a37266ada21d4a17b45bdee1225c7e1a P 39e2d0baa860401bbf588f9f9e03fef2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:45:30.420431 23520 catalog_manager.cc:5484] T a37266ada21d4a17b45bdee1225c7e1a P ff8bccc72e754c8cba77ead752f3fc4d reported cstate change: term changed from 0 to 1, leader changed from <none> to ff8bccc72e754c8cba77ead752f3fc4d (127.22.104.130). New cstate: current_term: 1 leader_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:30.420791 23822 raft_consensus.cc:348] T a37266ada21d4a17b45bdee1225c7e1a P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } } peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } }
I20240502 20:45:30.420876 23822 raft_consensus.cc:374] T a37266ada21d4a17b45bdee1225c7e1a P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.420899 23822 raft_consensus.cc:729] T a37266ada21d4a17b45bdee1225c7e1a P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39e2d0baa860401bbf588f9f9e03fef2, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.420992 23822 consensus_queue.cc:260] T a37266ada21d4a17b45bdee1225c7e1a P 39e2d0baa860401bbf588f9f9e03fef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } } peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } }
I20240502 20:45:30.421207 23822 ts_tablet_manager.cc:1398] T a37266ada21d4a17b45bdee1225c7e1a P 39e2d0baa860401bbf588f9f9e03fef2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:30.421347 23822 tablet_bootstrap.cc:492] T 49f3469a24a147c2a9b0f2634e2112e7 P 39e2d0baa860401bbf588f9f9e03fef2: Bootstrap starting.
I20240502 20:45:30.422191 23822 tablet_bootstrap.cc:654] T 49f3469a24a147c2a9b0f2634e2112e7 P 39e2d0baa860401bbf588f9f9e03fef2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.423341 23701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f3469a24a147c2a9b0f2634e2112e7" candidate_uuid: "cf520b51778147d08f5fa929459dfd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" is_pre_election: true
I20240502 20:45:30.423462 23701 raft_consensus.cc:2454] T 49f3469a24a147c2a9b0f2634e2112e7 P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf520b51778147d08f5fa929459dfd6e in term 0.
I20240502 20:45:30.423765 23822 tablet_bootstrap.cc:492] T 49f3469a24a147c2a9b0f2634e2112e7 P 39e2d0baa860401bbf588f9f9e03fef2: No bootstrap required, opened a new log
I20240502 20:45:30.423846 23590 leader_election.cc:302] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf520b51778147d08f5fa929459dfd6e, ff8bccc72e754c8cba77ead752f3fc4d; no voters:
I20240502 20:45:30.423957 23831 raft_consensus.cc:482] T a37266ada21d4a17b45bdee1225c7e1a P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:30.424014 23826 raft_consensus.cc:2781] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:30.424073 23826 raft_consensus.cc:482] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:30.424029 23831 raft_consensus.cc:504] T a37266ada21d4a17b45bdee1225c7e1a P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } } peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } }
I20240502 20:45:30.424479 23776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f3469a24a147c2a9b0f2634e2112e7" candidate_uuid: "cf520b51778147d08f5fa929459dfd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39e2d0baa860401bbf588f9f9e03fef2" is_pre_election: true
I20240502 20:45:30.424540 23831 leader_election.cc:288] T a37266ada21d4a17b45bdee1225c7e1a P 39e2d0baa860401bbf588f9f9e03fef2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf520b51778147d08f5fa929459dfd6e (127.22.104.129:34807), ff8bccc72e754c8cba77ead752f3fc4d (127.22.104.130:43443)
I20240502 20:45:30.424119 23826 raft_consensus.cc:3037] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Advancing to term 1
W20240502 20:45:30.425333 23590 leader_election.cc:341] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39e2d0baa860401bbf588f9f9e03fef2 (127.22.104.131:35163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 20:45:30.425742 23826 raft_consensus.cc:504] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.426069 23826 leader_election.cc:288] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e [CANDIDATE]: Term 1 election: Requested vote from peers ff8bccc72e754c8cba77ead752f3fc4d (127.22.104.130:43443), 39e2d0baa860401bbf588f9f9e03fef2 (127.22.104.131:35163)
I20240502 20:45:30.423849 23822 ts_tablet_manager.cc:1367] T 49f3469a24a147c2a9b0f2634e2112e7 P 39e2d0baa860401bbf588f9f9e03fef2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:45:30.426333 23776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f3469a24a147c2a9b0f2634e2112e7" candidate_uuid: "cf520b51778147d08f5fa929459dfd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39e2d0baa860401bbf588f9f9e03fef2"
I20240502 20:45:30.426352 23701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f3469a24a147c2a9b0f2634e2112e7" candidate_uuid: "cf520b51778147d08f5fa929459dfd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8bccc72e754c8cba77ead752f3fc4d"
I20240502 20:45:30.426486 23701 raft_consensus.cc:3037] T 49f3469a24a147c2a9b0f2634e2112e7 P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.426602 23822 raft_consensus.cc:348] T 49f3469a24a147c2a9b0f2634e2112e7 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.426678 23822 raft_consensus.cc:374] T 49f3469a24a147c2a9b0f2634e2112e7 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.426739 23822 raft_consensus.cc:729] T 49f3469a24a147c2a9b0f2634e2112e7 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39e2d0baa860401bbf588f9f9e03fef2, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.426846 23822 consensus_queue.cc:260] T 49f3469a24a147c2a9b0f2634e2112e7 P 39e2d0baa860401bbf588f9f9e03fef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.427096 23822 ts_tablet_manager.cc:1398] T 49f3469a24a147c2a9b0f2634e2112e7 P 39e2d0baa860401bbf588f9f9e03fef2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:45:30.427268 23822 tablet_bootstrap.cc:492] T 0fc3ca8a4b274046b1c1842318c8db21 P 39e2d0baa860401bbf588f9f9e03fef2: Bootstrap starting.
I20240502 20:45:30.427417 23701 raft_consensus.cc:2454] T 49f3469a24a147c2a9b0f2634e2112e7 P ff8bccc72e754c8cba77ead752f3fc4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf520b51778147d08f5fa929459dfd6e in term 1.
I20240502 20:45:30.428121 23822 tablet_bootstrap.cc:654] T 0fc3ca8a4b274046b1c1842318c8db21 P 39e2d0baa860401bbf588f9f9e03fef2: Neither blocks nor log segments found. Creating new log.
W20240502 20:45:30.426682 23590 leader_election.cc:341] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 39e2d0baa860401bbf588f9f9e03fef2 (127.22.104.131:35163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 20:45:30.429045 23624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37266ada21d4a17b45bdee1225c7e1a" candidate_uuid: "39e2d0baa860401bbf588f9f9e03fef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf520b51778147d08f5fa929459dfd6e" is_pre_election: true
I20240502 20:45:30.429190 23590 leader_election.cc:302] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf520b51778147d08f5fa929459dfd6e, ff8bccc72e754c8cba77ead752f3fc4d; no voters: 39e2d0baa860401bbf588f9f9e03fef2
I20240502 20:45:30.429226 23624 raft_consensus.cc:2379] T a37266ada21d4a17b45bdee1225c7e1a P cf520b51778147d08f5fa929459dfd6e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39e2d0baa860401bbf588f9f9e03fef2 in current term 1: Already voted for candidate ff8bccc72e754c8cba77ead752f3fc4d in this term.
I20240502 20:45:30.429333 23826 raft_consensus.cc:2781] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:30.429508 23826 raft_consensus.cc:686] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e [term 1 LEADER]: Becoming Leader. State: Replica: cf520b51778147d08f5fa929459dfd6e, State: Running, Role: LEADER
I20240502 20:45:30.429632 23826 consensus_queue.cc:237] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.429734 23701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37266ada21d4a17b45bdee1225c7e1a" candidate_uuid: "39e2d0baa860401bbf588f9f9e03fef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" is_pre_election: true
I20240502 20:45:30.430013 23742 leader_election.cc:302] T a37266ada21d4a17b45bdee1225c7e1a P 39e2d0baa860401bbf588f9f9e03fef2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39e2d0baa860401bbf588f9f9e03fef2; no voters: cf520b51778147d08f5fa929459dfd6e, ff8bccc72e754c8cba77ead752f3fc4d
I20240502 20:45:30.430152 23831 raft_consensus.cc:3037] T a37266ada21d4a17b45bdee1225c7e1a P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.430160 23822 tablet_bootstrap.cc:492] T 0fc3ca8a4b274046b1c1842318c8db21 P 39e2d0baa860401bbf588f9f9e03fef2: No bootstrap required, opened a new log
I20240502 20:45:30.430877 23822 ts_tablet_manager.cc:1367] T 0fc3ca8a4b274046b1c1842318c8db21 P 39e2d0baa860401bbf588f9f9e03fef2: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 20:45:30.431074 23831 raft_consensus.cc:2726] T a37266ada21d4a17b45bdee1225c7e1a P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 20:45:30.431174 23520 catalog_manager.cc:5484] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e reported cstate change: term changed from 0 to 1, leader changed from <none> to cf520b51778147d08f5fa929459dfd6e (127.22.104.129). New cstate: current_term: 1 leader_uuid: "cf520b51778147d08f5fa929459dfd6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:30.431288 23822 raft_consensus.cc:348] T 0fc3ca8a4b274046b1c1842318c8db21 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.431385 23822 raft_consensus.cc:374] T 0fc3ca8a4b274046b1c1842318c8db21 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.431416 23822 raft_consensus.cc:729] T 0fc3ca8a4b274046b1c1842318c8db21 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39e2d0baa860401bbf588f9f9e03fef2, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.431528 23822 consensus_queue.cc:260] T 0fc3ca8a4b274046b1c1842318c8db21 P 39e2d0baa860401bbf588f9f9e03fef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.431819 23822 ts_tablet_manager.cc:1398] T 0fc3ca8a4b274046b1c1842318c8db21 P 39e2d0baa860401bbf588f9f9e03fef2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:45:30.431874 23823 tablet_bootstrap.cc:492] T 13f3f643f725403eb35362f87f389145 P cf520b51778147d08f5fa929459dfd6e: No bootstrap required, opened a new log
I20240502 20:45:30.431947 23823 ts_tablet_manager.cc:1367] T 13f3f643f725403eb35362f87f389145 P cf520b51778147d08f5fa929459dfd6e: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240502 20:45:30.431998 23822 tablet_bootstrap.cc:492] T 1fa05319be264a8dacfba3fbb757629b P 39e2d0baa860401bbf588f9f9e03fef2: Bootstrap starting.
I20240502 20:45:30.432035 23827 tablet_bootstrap.cc:492] T 6f7bab52adb145df84de04cb1b7ea315 P ff8bccc72e754c8cba77ead752f3fc4d: No bootstrap required, opened a new log
I20240502 20:45:30.432101 23827 ts_tablet_manager.cc:1367] T 6f7bab52adb145df84de04cb1b7ea315 P ff8bccc72e754c8cba77ead752f3fc4d: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.001s
I20240502 20:45:30.432328 23823 raft_consensus.cc:348] T 13f3f643f725403eb35362f87f389145 P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.432416 23823 raft_consensus.cc:374] T 13f3f643f725403eb35362f87f389145 P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.432446 23823 raft_consensus.cc:729] T 13f3f643f725403eb35362f87f389145 P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf520b51778147d08f5fa929459dfd6e, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.432474 23827 raft_consensus.cc:348] T 6f7bab52adb145df84de04cb1b7ea315 P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.432562 23827 raft_consensus.cc:374] T 6f7bab52adb145df84de04cb1b7ea315 P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.432547 23823 consensus_queue.cc:260] T 13f3f643f725403eb35362f87f389145 P cf520b51778147d08f5fa929459dfd6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.432590 23827 raft_consensus.cc:729] T 6f7bab52adb145df84de04cb1b7ea315 P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff8bccc72e754c8cba77ead752f3fc4d, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.432718 23827 consensus_queue.cc:260] T 6f7bab52adb145df84de04cb1b7ea315 P ff8bccc72e754c8cba77ead752f3fc4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.432801 23823 ts_tablet_manager.cc:1398] T 13f3f643f725403eb35362f87f389145 P cf520b51778147d08f5fa929459dfd6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:30.432866 23822 tablet_bootstrap.cc:654] T 1fa05319be264a8dacfba3fbb757629b P 39e2d0baa860401bbf588f9f9e03fef2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.432953 23823 tablet_bootstrap.cc:492] T 1fa05319be264a8dacfba3fbb757629b P cf520b51778147d08f5fa929459dfd6e: Bootstrap starting.
I20240502 20:45:30.433802 23823 tablet_bootstrap.cc:654] T 1fa05319be264a8dacfba3fbb757629b P cf520b51778147d08f5fa929459dfd6e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.433986 23822 tablet_bootstrap.cc:492] T 1fa05319be264a8dacfba3fbb757629b P 39e2d0baa860401bbf588f9f9e03fef2: No bootstrap required, opened a new log
I20240502 20:45:30.434099 23822 ts_tablet_manager.cc:1367] T 1fa05319be264a8dacfba3fbb757629b P 39e2d0baa860401bbf588f9f9e03fef2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 20:45:30.434506 23822 raft_consensus.cc:348] T 1fa05319be264a8dacfba3fbb757629b P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.434654 23822 raft_consensus.cc:374] T 1fa05319be264a8dacfba3fbb757629b P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.434762 23822 raft_consensus.cc:729] T 1fa05319be264a8dacfba3fbb757629b P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39e2d0baa860401bbf588f9f9e03fef2, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.434808 23823 tablet_bootstrap.cc:492] T 1fa05319be264a8dacfba3fbb757629b P cf520b51778147d08f5fa929459dfd6e: No bootstrap required, opened a new log
I20240502 20:45:30.434918 23823 ts_tablet_manager.cc:1367] T 1fa05319be264a8dacfba3fbb757629b P cf520b51778147d08f5fa929459dfd6e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:30.434964 23822 consensus_queue.cc:260] T 1fa05319be264a8dacfba3fbb757629b P 39e2d0baa860401bbf588f9f9e03fef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.432955 23827 ts_tablet_manager.cc:1398] T 6f7bab52adb145df84de04cb1b7ea315 P ff8bccc72e754c8cba77ead752f3fc4d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:45:30.435221 23822 ts_tablet_manager.cc:1398] T 1fa05319be264a8dacfba3fbb757629b P 39e2d0baa860401bbf588f9f9e03fef2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:45:30.435289 23827 tablet_bootstrap.cc:492] T 74948efa3e364ee596ea1050cddaaae2 P ff8bccc72e754c8cba77ead752f3fc4d: Bootstrap starting.
I20240502 20:45:30.435340 23822 tablet_bootstrap.cc:492] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2: Bootstrap starting.
I20240502 20:45:30.435333 23823 raft_consensus.cc:348] T 1fa05319be264a8dacfba3fbb757629b P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.435437 23823 raft_consensus.cc:374] T 1fa05319be264a8dacfba3fbb757629b P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.435470 23823 raft_consensus.cc:729] T 1fa05319be264a8dacfba3fbb757629b P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf520b51778147d08f5fa929459dfd6e, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.435560 23823 consensus_queue.cc:260] T 1fa05319be264a8dacfba3fbb757629b P cf520b51778147d08f5fa929459dfd6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.435815 23823 ts_tablet_manager.cc:1398] T 1fa05319be264a8dacfba3fbb757629b P cf520b51778147d08f5fa929459dfd6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:30.436172 23822 tablet_bootstrap.cc:654] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.436172 23827 tablet_bootstrap.cc:654] T 74948efa3e364ee596ea1050cddaaae2 P ff8bccc72e754c8cba77ead752f3fc4d: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.437215 23823 tablet_bootstrap.cc:492] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e: Bootstrap starting.
I20240502 20:45:30.438105 23847 raft_consensus.cc:482] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:30.438187 23847 raft_consensus.cc:504] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.438377 23823 tablet_bootstrap.cc:654] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.438560 23847 leader_election.cc:288] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf520b51778147d08f5fa929459dfd6e (127.22.104.129:34807), 39e2d0baa860401bbf588f9f9e03fef2 (127.22.104.131:35163)
I20240502 20:45:30.439059 23624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa05319be264a8dacfba3fbb757629b" candidate_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf520b51778147d08f5fa929459dfd6e" is_pre_election: true
I20240502 20:45:30.439170 23624 raft_consensus.cc:2454] T 1fa05319be264a8dacfba3fbb757629b P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff8bccc72e754c8cba77ead752f3fc4d in term 0.
I20240502 20:45:30.439505 23823 tablet_bootstrap.cc:492] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e: No bootstrap required, opened a new log
I20240502 20:45:30.439594 23823 ts_tablet_manager.cc:1367] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:30.439579 23776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa05319be264a8dacfba3fbb757629b" candidate_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39e2d0baa860401bbf588f9f9e03fef2" is_pre_election: true
I20240502 20:45:30.439716 23776 raft_consensus.cc:2454] T 1fa05319be264a8dacfba3fbb757629b P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff8bccc72e754c8cba77ead752f3fc4d in term 0.
I20240502 20:45:30.439931 23667 leader_election.cc:302] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39e2d0baa860401bbf588f9f9e03fef2, ff8bccc72e754c8cba77ead752f3fc4d; no voters:
I20240502 20:45:30.440011 23823 raft_consensus.cc:348] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.440097 23823 raft_consensus.cc:374] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.440110 23847 raft_consensus.cc:2781] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:30.440132 23823 raft_consensus.cc:729] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf520b51778147d08f5fa929459dfd6e, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.440152 23847 raft_consensus.cc:482] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:30.440188 23847 raft_consensus.cc:3037] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.440248 23823 consensus_queue.cc:260] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.440490 23823 ts_tablet_manager.cc:1398] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:30.440649 23823 tablet_bootstrap.cc:492] T 6f7bab52adb145df84de04cb1b7ea315 P cf520b51778147d08f5fa929459dfd6e: Bootstrap starting.
I20240502 20:45:30.441164 23847 raft_consensus.cc:504] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.441468 23823 tablet_bootstrap.cc:654] T 6f7bab52adb145df84de04cb1b7ea315 P cf520b51778147d08f5fa929459dfd6e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.441471 23847 leader_election.cc:288] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d [CANDIDATE]: Term 1 election: Requested vote from peers cf520b51778147d08f5fa929459dfd6e (127.22.104.129:34807), 39e2d0baa860401bbf588f9f9e03fef2 (127.22.104.131:35163)
I20240502 20:45:30.441869 23776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa05319be264a8dacfba3fbb757629b" candidate_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39e2d0baa860401bbf588f9f9e03fef2"
I20240502 20:45:30.441977 23776 raft_consensus.cc:3037] T 1fa05319be264a8dacfba3fbb757629b P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.441849 23624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa05319be264a8dacfba3fbb757629b" candidate_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf520b51778147d08f5fa929459dfd6e"
I20240502 20:45:30.442092 23624 raft_consensus.cc:3037] T 1fa05319be264a8dacfba3fbb757629b P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.442608 23823 tablet_bootstrap.cc:492] T 6f7bab52adb145df84de04cb1b7ea315 P cf520b51778147d08f5fa929459dfd6e: No bootstrap required, opened a new log
I20240502 20:45:30.442687 23823 ts_tablet_manager.cc:1367] T 6f7bab52adb145df84de04cb1b7ea315 P cf520b51778147d08f5fa929459dfd6e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:30.442873 23624 raft_consensus.cc:2454] T 1fa05319be264a8dacfba3fbb757629b P cf520b51778147d08f5fa929459dfd6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff8bccc72e754c8cba77ead752f3fc4d in term 1.
I20240502 20:45:30.442873 23776 raft_consensus.cc:2454] T 1fa05319be264a8dacfba3fbb757629b P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff8bccc72e754c8cba77ead752f3fc4d in term 1.
I20240502 20:45:30.443025 23823 raft_consensus.cc:348] T 6f7bab52adb145df84de04cb1b7ea315 P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.443121 23823 raft_consensus.cc:374] T 6f7bab52adb145df84de04cb1b7ea315 P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.443154 23823 raft_consensus.cc:729] T 6f7bab52adb145df84de04cb1b7ea315 P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf520b51778147d08f5fa929459dfd6e, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.443207 23667 leader_election.cc:302] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39e2d0baa860401bbf588f9f9e03fef2, ff8bccc72e754c8cba77ead752f3fc4d; no voters:
I20240502 20:45:30.443256 23823 consensus_queue.cc:260] T 6f7bab52adb145df84de04cb1b7ea315 P cf520b51778147d08f5fa929459dfd6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.443492 23823 ts_tablet_manager.cc:1398] T 6f7bab52adb145df84de04cb1b7ea315 P cf520b51778147d08f5fa929459dfd6e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:45:30.443532 23847 raft_consensus.cc:2781] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:30.443583 23827 tablet_bootstrap.cc:492] T 74948efa3e364ee596ea1050cddaaae2 P ff8bccc72e754c8cba77ead752f3fc4d: No bootstrap required, opened a new log
I20240502 20:45:30.443616 23847 raft_consensus.cc:686] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d [term 1 LEADER]: Becoming Leader. State: Replica: ff8bccc72e754c8cba77ead752f3fc4d, State: Running, Role: LEADER
I20240502 20:45:30.443657 23827 ts_tablet_manager.cc:1367] T 74948efa3e364ee596ea1050cddaaae2 P ff8bccc72e754c8cba77ead752f3fc4d: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240502 20:45:30.443729 23847 consensus_queue.cc:237] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.444146 23827 raft_consensus.cc:348] T 74948efa3e364ee596ea1050cddaaae2 P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.444244 23827 raft_consensus.cc:374] T 74948efa3e364ee596ea1050cddaaae2 P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.444272 23827 raft_consensus.cc:729] T 74948efa3e364ee596ea1050cddaaae2 P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff8bccc72e754c8cba77ead752f3fc4d, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.444401 23827 consensus_queue.cc:260] T 74948efa3e364ee596ea1050cddaaae2 P ff8bccc72e754c8cba77ead752f3fc4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.444656 23827 ts_tablet_manager.cc:1398] T 74948efa3e364ee596ea1050cddaaae2 P ff8bccc72e754c8cba77ead752f3fc4d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:45:30.444813 23827 tablet_bootstrap.cc:492] T 1ee72890ba03492b9eebab393765739f P ff8bccc72e754c8cba77ead752f3fc4d: Bootstrap starting.
I20240502 20:45:30.445696 23827 tablet_bootstrap.cc:654] T 1ee72890ba03492b9eebab393765739f P ff8bccc72e754c8cba77ead752f3fc4d: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.446557 23827 tablet_bootstrap.cc:492] T 1ee72890ba03492b9eebab393765739f P ff8bccc72e754c8cba77ead752f3fc4d: No bootstrap required, opened a new log
I20240502 20:45:30.446646 23827 ts_tablet_manager.cc:1367] T 1ee72890ba03492b9eebab393765739f P ff8bccc72e754c8cba77ead752f3fc4d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:45:30.447036 23827 raft_consensus.cc:348] T 1ee72890ba03492b9eebab393765739f P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.447120 23827 raft_consensus.cc:374] T 1ee72890ba03492b9eebab393765739f P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.447217 23827 raft_consensus.cc:729] T 1ee72890ba03492b9eebab393765739f P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff8bccc72e754c8cba77ead752f3fc4d, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.447108 23517 catalog_manager.cc:5484] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d reported cstate change: term changed from 0 to 1, leader changed from <none> to ff8bccc72e754c8cba77ead752f3fc4d (127.22.104.130). New cstate: current_term: 1 leader_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:30.447397 23827 consensus_queue.cc:260] T 1ee72890ba03492b9eebab393765739f P ff8bccc72e754c8cba77ead752f3fc4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.447635 23827 ts_tablet_manager.cc:1398] T 1ee72890ba03492b9eebab393765739f P ff8bccc72e754c8cba77ead752f3fc4d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:45:30.448310 23822 tablet_bootstrap.cc:492] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2: No bootstrap required, opened a new log
I20240502 20:45:30.448407 23822 ts_tablet_manager.cc:1367] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.001s
I20240502 20:45:30.448786 23822 raft_consensus.cc:348] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.448909 23822 raft_consensus.cc:374] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.448964 23822 raft_consensus.cc:729] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39e2d0baa860401bbf588f9f9e03fef2, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.449095 23822 consensus_queue.cc:260] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.449364 23822 ts_tablet_manager.cc:1398] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:45:30.490859 23831 raft_consensus.cc:482] T 0fc3ca8a4b274046b1c1842318c8db21 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:30.491016 23831 raft_consensus.cc:504] T 0fc3ca8a4b274046b1c1842318c8db21 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.491453 23831 leader_election.cc:288] T 0fc3ca8a4b274046b1c1842318c8db21 P 39e2d0baa860401bbf588f9f9e03fef2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf520b51778147d08f5fa929459dfd6e (127.22.104.129:34807), ff8bccc72e754c8cba77ead752f3fc4d (127.22.104.130:43443)
I20240502 20:45:30.491915 23701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc3ca8a4b274046b1c1842318c8db21" candidate_uuid: "39e2d0baa860401bbf588f9f9e03fef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" is_pre_election: true
I20240502 20:45:30.491916 23624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc3ca8a4b274046b1c1842318c8db21" candidate_uuid: "39e2d0baa860401bbf588f9f9e03fef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf520b51778147d08f5fa929459dfd6e" is_pre_election: true
I20240502 20:45:30.492059 23701 raft_consensus.cc:2454] T 0fc3ca8a4b274046b1c1842318c8db21 P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39e2d0baa860401bbf588f9f9e03fef2 in term 0.
I20240502 20:45:30.492125 23624 raft_consensus.cc:2454] T 0fc3ca8a4b274046b1c1842318c8db21 P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39e2d0baa860401bbf588f9f9e03fef2 in term 0.
I20240502 20:45:30.492343 23742 leader_election.cc:302] T 0fc3ca8a4b274046b1c1842318c8db21 P 39e2d0baa860401bbf588f9f9e03fef2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39e2d0baa860401bbf588f9f9e03fef2, ff8bccc72e754c8cba77ead752f3fc4d; no voters:
I20240502 20:45:30.492511 23831 raft_consensus.cc:2781] T 0fc3ca8a4b274046b1c1842318c8db21 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:30.492570 23831 raft_consensus.cc:482] T 0fc3ca8a4b274046b1c1842318c8db21 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:30.492616 23831 raft_consensus.cc:3037] T 0fc3ca8a4b274046b1c1842318c8db21 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.493409 23831 raft_consensus.cc:504] T 0fc3ca8a4b274046b1c1842318c8db21 P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.493767 23831 leader_election.cc:288] T 0fc3ca8a4b274046b1c1842318c8db21 P 39e2d0baa860401bbf588f9f9e03fef2 [CANDIDATE]: Term 1 election: Requested vote from peers cf520b51778147d08f5fa929459dfd6e (127.22.104.129:34807), ff8bccc72e754c8cba77ead752f3fc4d (127.22.104.130:43443)
I20240502 20:45:30.494094 23701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc3ca8a4b274046b1c1842318c8db21" candidate_uuid: "39e2d0baa860401bbf588f9f9e03fef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8bccc72e754c8cba77ead752f3fc4d"
I20240502 20:45:30.494098 23624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc3ca8a4b274046b1c1842318c8db21" candidate_uuid: "39e2d0baa860401bbf588f9f9e03fef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf520b51778147d08f5fa929459dfd6e"
I20240502 20:45:30.494217 23624 raft_consensus.cc:3037] T 0fc3ca8a4b274046b1c1842318c8db21 P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.494217 23701 raft_consensus.cc:3037] T 0fc3ca8a4b274046b1c1842318c8db21 P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.495137 23624 raft_consensus.cc:2454] T 0fc3ca8a4b274046b1c1842318c8db21 P cf520b51778147d08f5fa929459dfd6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39e2d0baa860401bbf588f9f9e03fef2 in term 1.
I20240502 20:45:30.495137 23701 raft_consensus.cc:2454] T 0fc3ca8a4b274046b1c1842318c8db21 P ff8bccc72e754c8cba77ead752f3fc4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39e2d0baa860401bbf588f9f9e03fef2 in term 1.
I20240502 20:45:30.495435 23742 leader_election.cc:302] T 0fc3ca8a4b274046b1c1842318c8db21 P 39e2d0baa860401bbf588f9f9e03fef2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39e2d0baa860401bbf588f9f9e03fef2, cf520b51778147d08f5fa929459dfd6e; no voters:
I20240502 20:45:30.495584 23831 raft_consensus.cc:2781] T 0fc3ca8a4b274046b1c1842318c8db21 P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:30.495788 23831 raft_consensus.cc:686] T 0fc3ca8a4b274046b1c1842318c8db21 P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 LEADER]: Becoming Leader. State: Replica: 39e2d0baa860401bbf588f9f9e03fef2, State: Running, Role: LEADER
I20240502 20:45:30.495946 23831 consensus_queue.cc:237] T 0fc3ca8a4b274046b1c1842318c8db21 P 39e2d0baa860401bbf588f9f9e03fef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.497469 23517 catalog_manager.cc:5484] T 0fc3ca8a4b274046b1c1842318c8db21 P 39e2d0baa860401bbf588f9f9e03fef2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39e2d0baa860401bbf588f9f9e03fef2 (127.22.104.131). New cstate: current_term: 1 leader_uuid: "39e2d0baa860401bbf588f9f9e03fef2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:30.510083 23855 raft_consensus.cc:482] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:30.510190 23855 raft_consensus.cc:504] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.510509 23855 leader_election.cc:288] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff8bccc72e754c8cba77ead752f3fc4d (127.22.104.130:43443), 39e2d0baa860401bbf588f9f9e03fef2 (127.22.104.131:35163)
I20240502 20:45:30.510895 23701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee72890ba03492b9eebab393765739f" candidate_uuid: "cf520b51778147d08f5fa929459dfd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" is_pre_election: true
I20240502 20:45:30.510895 23776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee72890ba03492b9eebab393765739f" candidate_uuid: "cf520b51778147d08f5fa929459dfd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39e2d0baa860401bbf588f9f9e03fef2" is_pre_election: true
I20240502 20:45:30.511047 23776 raft_consensus.cc:2454] T 1ee72890ba03492b9eebab393765739f P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf520b51778147d08f5fa929459dfd6e in term 0.
I20240502 20:45:30.511052 23701 raft_consensus.cc:2454] T 1ee72890ba03492b9eebab393765739f P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf520b51778147d08f5fa929459dfd6e in term 0.
I20240502 20:45:30.511327 23590 leader_election.cc:302] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39e2d0baa860401bbf588f9f9e03fef2, cf520b51778147d08f5fa929459dfd6e; no voters:
I20240502 20:45:30.511498 23855 raft_consensus.cc:2781] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:30.511566 23855 raft_consensus.cc:482] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:30.511592 23855 raft_consensus.cc:3037] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.512275 23826 raft_consensus.cc:482] T 74948efa3e364ee596ea1050cddaaae2 P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:30.512369 23826 raft_consensus.cc:504] T 74948efa3e364ee596ea1050cddaaae2 P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.512392 23855 raft_consensus.cc:504] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.512681 23855 leader_election.cc:288] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e [CANDIDATE]: Term 1 election: Requested vote from peers ff8bccc72e754c8cba77ead752f3fc4d (127.22.104.130:43443), 39e2d0baa860401bbf588f9f9e03fef2 (127.22.104.131:35163)
I20240502 20:45:30.512693 23826 leader_election.cc:288] T 74948efa3e364ee596ea1050cddaaae2 P cf520b51778147d08f5fa929459dfd6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff8bccc72e754c8cba77ead752f3fc4d (127.22.104.130:43443), 39e2d0baa860401bbf588f9f9e03fef2 (127.22.104.131:35163)
I20240502 20:45:30.512967 23701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74948efa3e364ee596ea1050cddaaae2" candidate_uuid: "cf520b51778147d08f5fa929459dfd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" is_pre_election: true
I20240502 20:45:30.513069 23700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee72890ba03492b9eebab393765739f" candidate_uuid: "cf520b51778147d08f5fa929459dfd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8bccc72e754c8cba77ead752f3fc4d"
I20240502 20:45:30.513113 23701 raft_consensus.cc:2454] T 74948efa3e364ee596ea1050cddaaae2 P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf520b51778147d08f5fa929459dfd6e in term 0.
I20240502 20:45:30.513185 23700 raft_consensus.cc:3037] T 1ee72890ba03492b9eebab393765739f P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.513284 23776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee72890ba03492b9eebab393765739f" candidate_uuid: "cf520b51778147d08f5fa929459dfd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39e2d0baa860401bbf588f9f9e03fef2"
I20240502 20:45:30.513372 23776 raft_consensus.cc:3037] T 1ee72890ba03492b9eebab393765739f P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.513408 23590 leader_election.cc:302] T 74948efa3e364ee596ea1050cddaaae2 P cf520b51778147d08f5fa929459dfd6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf520b51778147d08f5fa929459dfd6e, ff8bccc72e754c8cba77ead752f3fc4d; no voters:
I20240502 20:45:30.513590 23826 raft_consensus.cc:2781] T 74948efa3e364ee596ea1050cddaaae2 P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:30.513646 23826 raft_consensus.cc:482] T 74948efa3e364ee596ea1050cddaaae2 P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:30.513679 23826 raft_consensus.cc:3037] T 74948efa3e364ee596ea1050cddaaae2 P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.514120 23700 raft_consensus.cc:2454] T 1ee72890ba03492b9eebab393765739f P ff8bccc72e754c8cba77ead752f3fc4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf520b51778147d08f5fa929459dfd6e in term 1.
I20240502 20:45:30.513157 23775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74948efa3e364ee596ea1050cddaaae2" candidate_uuid: "cf520b51778147d08f5fa929459dfd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39e2d0baa860401bbf588f9f9e03fef2" is_pre_election: true
I20240502 20:45:30.514284 23775 raft_consensus.cc:2454] T 74948efa3e364ee596ea1050cddaaae2 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf520b51778147d08f5fa929459dfd6e in term 0.
I20240502 20:45:30.514330 23776 raft_consensus.cc:2454] T 1ee72890ba03492b9eebab393765739f P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf520b51778147d08f5fa929459dfd6e in term 1.
I20240502 20:45:30.514382 23590 leader_election.cc:302] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf520b51778147d08f5fa929459dfd6e, ff8bccc72e754c8cba77ead752f3fc4d; no voters:
I20240502 20:45:30.514379 23826 raft_consensus.cc:504] T 74948efa3e364ee596ea1050cddaaae2 P cf520b51778147d08f5fa929459dfd6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
W20240502 20:45:30.514825 23651 tablet.cc:2369] T 0fc3ca8a4b274046b1c1842318c8db21 P cf520b51778147d08f5fa929459dfd6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:45:30.514922 23826 leader_election.cc:288] T 74948efa3e364ee596ea1050cddaaae2 P cf520b51778147d08f5fa929459dfd6e [CANDIDATE]: Term 1 election: Requested vote from peers ff8bccc72e754c8cba77ead752f3fc4d (127.22.104.130:43443), 39e2d0baa860401bbf588f9f9e03fef2 (127.22.104.131:35163)
I20240502 20:45:30.515019 23826 raft_consensus.cc:2781] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:30.515075 23826 raft_consensus.cc:686] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e [term 1 LEADER]: Becoming Leader. State: Replica: cf520b51778147d08f5fa929459dfd6e, State: Running, Role: LEADER
I20240502 20:45:30.515205 23826 consensus_queue.cc:237] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.515269 23700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74948efa3e364ee596ea1050cddaaae2" candidate_uuid: "cf520b51778147d08f5fa929459dfd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8bccc72e754c8cba77ead752f3fc4d"
I20240502 20:45:30.515338 23700 raft_consensus.cc:3037] T 74948efa3e364ee596ea1050cddaaae2 P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.515391 23776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74948efa3e364ee596ea1050cddaaae2" candidate_uuid: "cf520b51778147d08f5fa929459dfd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39e2d0baa860401bbf588f9f9e03fef2"
I20240502 20:45:30.515472 23776 raft_consensus.cc:3037] T 74948efa3e364ee596ea1050cddaaae2 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.516101 23700 raft_consensus.cc:2454] T 74948efa3e364ee596ea1050cddaaae2 P ff8bccc72e754c8cba77ead752f3fc4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf520b51778147d08f5fa929459dfd6e in term 1.
I20240502 20:45:30.516119 23776 raft_consensus.cc:2454] T 74948efa3e364ee596ea1050cddaaae2 P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf520b51778147d08f5fa929459dfd6e in term 1.
I20240502 20:45:30.516351 23590 leader_election.cc:302] T 74948efa3e364ee596ea1050cddaaae2 P cf520b51778147d08f5fa929459dfd6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39e2d0baa860401bbf588f9f9e03fef2, cf520b51778147d08f5fa929459dfd6e; no voters:
I20240502 20:45:30.516498 23855 raft_consensus.cc:2781] T 74948efa3e364ee596ea1050cddaaae2 P cf520b51778147d08f5fa929459dfd6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:30.516578 23855 raft_consensus.cc:686] T 74948efa3e364ee596ea1050cddaaae2 P cf520b51778147d08f5fa929459dfd6e [term 1 LEADER]: Becoming Leader. State: Replica: cf520b51778147d08f5fa929459dfd6e, State: Running, Role: LEADER
I20240502 20:45:30.516690 23855 consensus_queue.cc:237] T 74948efa3e364ee596ea1050cddaaae2 P cf520b51778147d08f5fa929459dfd6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.516903 23517 catalog_manager.cc:5484] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e reported cstate change: term changed from 0 to 1, leader changed from <none> to cf520b51778147d08f5fa929459dfd6e (127.22.104.129). New cstate: current_term: 1 leader_uuid: "cf520b51778147d08f5fa929459dfd6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:30.519692 23517 catalog_manager.cc:5484] T 74948efa3e364ee596ea1050cddaaae2 P cf520b51778147d08f5fa929459dfd6e reported cstate change: term changed from 0 to 1, leader changed from <none> to cf520b51778147d08f5fa929459dfd6e (127.22.104.129). New cstate: current_term: 1 leader_uuid: "cf520b51778147d08f5fa929459dfd6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:30.520561 23831 raft_consensus.cc:482] T 6f7bab52adb145df84de04cb1b7ea315 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:30.520637 23831 raft_consensus.cc:504] T 6f7bab52adb145df84de04cb1b7ea315 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.520931 23831 leader_election.cc:288] T 6f7bab52adb145df84de04cb1b7ea315 P 39e2d0baa860401bbf588f9f9e03fef2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf520b51778147d08f5fa929459dfd6e (127.22.104.129:34807), ff8bccc72e754c8cba77ead752f3fc4d (127.22.104.130:43443)
I20240502 20:45:30.521188 23624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f7bab52adb145df84de04cb1b7ea315" candidate_uuid: "39e2d0baa860401bbf588f9f9e03fef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf520b51778147d08f5fa929459dfd6e" is_pre_election: true
I20240502 20:45:30.521230 23700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f7bab52adb145df84de04cb1b7ea315" candidate_uuid: "39e2d0baa860401bbf588f9f9e03fef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" is_pre_election: true
I20240502 20:45:30.521297 23624 raft_consensus.cc:2454] T 6f7bab52adb145df84de04cb1b7ea315 P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39e2d0baa860401bbf588f9f9e03fef2 in term 0.
I20240502 20:45:30.521297 23700 raft_consensus.cc:2454] T 6f7bab52adb145df84de04cb1b7ea315 P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39e2d0baa860401bbf588f9f9e03fef2 in term 0.
I20240502 20:45:30.521567 23742 leader_election.cc:302] T 6f7bab52adb145df84de04cb1b7ea315 P 39e2d0baa860401bbf588f9f9e03fef2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39e2d0baa860401bbf588f9f9e03fef2, ff8bccc72e754c8cba77ead752f3fc4d; no voters:
I20240502 20:45:30.521733 23831 raft_consensus.cc:2781] T 6f7bab52adb145df84de04cb1b7ea315 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:30.521778 23831 raft_consensus.cc:482] T 6f7bab52adb145df84de04cb1b7ea315 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:30.521813 23831 raft_consensus.cc:3037] T 6f7bab52adb145df84de04cb1b7ea315 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.522416 23831 raft_consensus.cc:504] T 6f7bab52adb145df84de04cb1b7ea315 P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.522687 23831 leader_election.cc:288] T 6f7bab52adb145df84de04cb1b7ea315 P 39e2d0baa860401bbf588f9f9e03fef2 [CANDIDATE]: Term 1 election: Requested vote from peers cf520b51778147d08f5fa929459dfd6e (127.22.104.129:34807), ff8bccc72e754c8cba77ead752f3fc4d (127.22.104.130:43443)
I20240502 20:45:30.523002 23624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f7bab52adb145df84de04cb1b7ea315" candidate_uuid: "39e2d0baa860401bbf588f9f9e03fef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf520b51778147d08f5fa929459dfd6e"
I20240502 20:45:30.523067 23700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f7bab52adb145df84de04cb1b7ea315" candidate_uuid: "39e2d0baa860401bbf588f9f9e03fef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8bccc72e754c8cba77ead752f3fc4d"
I20240502 20:45:30.523097 23624 raft_consensus.cc:3037] T 6f7bab52adb145df84de04cb1b7ea315 P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.523170 23700 raft_consensus.cc:3037] T 6f7bab52adb145df84de04cb1b7ea315 P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.523746 23624 raft_consensus.cc:2454] T 6f7bab52adb145df84de04cb1b7ea315 P cf520b51778147d08f5fa929459dfd6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39e2d0baa860401bbf588f9f9e03fef2 in term 1.
I20240502 20:45:30.523757 23700 raft_consensus.cc:2454] T 6f7bab52adb145df84de04cb1b7ea315 P ff8bccc72e754c8cba77ead752f3fc4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39e2d0baa860401bbf588f9f9e03fef2 in term 1.
I20240502 20:45:30.524003 23742 leader_election.cc:302] T 6f7bab52adb145df84de04cb1b7ea315 P 39e2d0baa860401bbf588f9f9e03fef2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39e2d0baa860401bbf588f9f9e03fef2, cf520b51778147d08f5fa929459dfd6e; no voters:
I20240502 20:45:30.524156 23831 raft_consensus.cc:2781] T 6f7bab52adb145df84de04cb1b7ea315 P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:30.524236 23831 raft_consensus.cc:686] T 6f7bab52adb145df84de04cb1b7ea315 P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 LEADER]: Becoming Leader. State: Replica: 39e2d0baa860401bbf588f9f9e03fef2, State: Running, Role: LEADER
I20240502 20:45:30.524371 23831 consensus_queue.cc:237] T 6f7bab52adb145df84de04cb1b7ea315 P 39e2d0baa860401bbf588f9f9e03fef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.525761 23517 catalog_manager.cc:5484] T 6f7bab52adb145df84de04cb1b7ea315 P 39e2d0baa860401bbf588f9f9e03fef2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39e2d0baa860401bbf588f9f9e03fef2 (127.22.104.131). New cstate: current_term: 1 leader_uuid: "39e2d0baa860401bbf588f9f9e03fef2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:30.535880 23831 raft_consensus.cc:482] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:30.535969 23831 raft_consensus.cc:504] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.536260 23831 leader_election.cc:288] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf520b51778147d08f5fa929459dfd6e (127.22.104.129:34807), ff8bccc72e754c8cba77ead752f3fc4d (127.22.104.130:43443)
I20240502 20:45:30.536562 23624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13f3f643f725403eb35362f87f389145" candidate_uuid: "39e2d0baa860401bbf588f9f9e03fef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf520b51778147d08f5fa929459dfd6e" is_pre_election: true
I20240502 20:45:30.536612 23700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13f3f643f725403eb35362f87f389145" candidate_uuid: "39e2d0baa860401bbf588f9f9e03fef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" is_pre_election: true
I20240502 20:45:30.536664 23624 raft_consensus.cc:2454] T 13f3f643f725403eb35362f87f389145 P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39e2d0baa860401bbf588f9f9e03fef2 in term 0.
I20240502 20:45:30.536708 23700 raft_consensus.cc:2454] T 13f3f643f725403eb35362f87f389145 P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39e2d0baa860401bbf588f9f9e03fef2 in term 0.
I20240502 20:45:30.536916 23742 leader_election.cc:302] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39e2d0baa860401bbf588f9f9e03fef2, cf520b51778147d08f5fa929459dfd6e; no voters:
I20240502 20:45:30.537086 23831 raft_consensus.cc:2781] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:30.537139 23831 raft_consensus.cc:482] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:30.537165 23831 raft_consensus.cc:3037] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.537838 23831 raft_consensus.cc:504] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.538110 23831 leader_election.cc:288] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2 [CANDIDATE]: Term 1 election: Requested vote from peers cf520b51778147d08f5fa929459dfd6e (127.22.104.129:34807), ff8bccc72e754c8cba77ead752f3fc4d (127.22.104.130:43443)
I20240502 20:45:30.538394 23624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13f3f643f725403eb35362f87f389145" candidate_uuid: "39e2d0baa860401bbf588f9f9e03fef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf520b51778147d08f5fa929459dfd6e"
I20240502 20:45:30.538421 23700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13f3f643f725403eb35362f87f389145" candidate_uuid: "39e2d0baa860401bbf588f9f9e03fef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8bccc72e754c8cba77ead752f3fc4d"
I20240502 20:45:30.538511 23624 raft_consensus.cc:3037] T 13f3f643f725403eb35362f87f389145 P cf520b51778147d08f5fa929459dfd6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.538517 23700 raft_consensus.cc:3037] T 13f3f643f725403eb35362f87f389145 P ff8bccc72e754c8cba77ead752f3fc4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.539186 23700 raft_consensus.cc:2454] T 13f3f643f725403eb35362f87f389145 P ff8bccc72e754c8cba77ead752f3fc4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39e2d0baa860401bbf588f9f9e03fef2 in term 1.
I20240502 20:45:30.539191 23624 raft_consensus.cc:2454] T 13f3f643f725403eb35362f87f389145 P cf520b51778147d08f5fa929459dfd6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39e2d0baa860401bbf588f9f9e03fef2 in term 1.
I20240502 20:45:30.539455 23742 leader_election.cc:302] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39e2d0baa860401bbf588f9f9e03fef2, cf520b51778147d08f5fa929459dfd6e; no voters:
I20240502 20:45:30.539626 23831 raft_consensus.cc:2781] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:30.539701 23831 raft_consensus.cc:686] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 LEADER]: Becoming Leader. State: Replica: 39e2d0baa860401bbf588f9f9e03fef2, State: Running, Role: LEADER
I20240502 20:45:30.539821 23831 consensus_queue.cc:237] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.541153 23517 catalog_manager.cc:5484] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39e2d0baa860401bbf588f9f9e03fef2 (127.22.104.131). New cstate: current_term: 1 leader_uuid: "39e2d0baa860401bbf588f9f9e03fef2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:30.562928 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:30.564854 23871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:30.565260 23872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:30.565596 23874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:30.566205 22946 server_base.cc:1034] running on GCE node
I20240502 20:45:30.566371 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:45:30.566407 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:45:30.566427 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682730566427 us; error 0 us; skew 500 ppm
I20240502 20:45:30.567433 22946 webserver.cc:460] Webserver started at http://127.22.104.132:34229/ using document root <none> and password file <none>
I20240502 20:45:30.567575 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:30.567621 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:30.567692 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:30.568058 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714682726530369-22946-0/minicluster-data/ts-3-root/instance:
uuid: "7e5c2bf991e945d7a845d292c7fc9b77"
format_stamp: "Formatted at 2024-05-02 20:45:30 on dist-test-slave-rgns"
I20240502 20:45:30.569402 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 20:45:30.570261 23879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:30.570483 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 20:45:30.570549 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714682726530369-22946-0/minicluster-data/ts-3-root
uuid: "7e5c2bf991e945d7a845d292c7fc9b77"
format_stamp: "Formatted at 2024-05-02 20:45:30 on dist-test-slave-rgns"
I20240502 20:45:30.570616 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714682726530369-22946-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714682726530369-22946-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714682726530369-22946-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:30.575484 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:30.575793 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:30.576053 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:30.576467 22946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:30.576514 22946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:30.576548 22946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:30.576576 22946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:30.581367 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.132:33565
I20240502 20:45:30.581498 23943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.132:33565 every 8 connection(s)
I20240502 20:45:30.585466 23944 heartbeater.cc:347] Connected to a master server at 127.22.104.190:32947
I20240502 20:45:30.585563 23944 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:30.585776 23944 heartbeater.cc:511] Master 127.22.104.190:32947 requested a full tablet report, sending...
I20240502 20:45:30.586372 23517 ts_manager.cc:194] Registered new tserver with Master: 7e5c2bf991e945d7a845d292c7fc9b77 (127.22.104.132:33565)
I20240502 20:45:30.586951 23517 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48996
I20240502 20:45:30.709609 23831 raft_consensus.cc:482] T 49f3469a24a147c2a9b0f2634e2112e7 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:30.709777 23831 raft_consensus.cc:504] T 49f3469a24a147c2a9b0f2634e2112e7 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } }
I20240502 20:45:30.710222 23831 leader_election.cc:288] T 49f3469a24a147c2a9b0f2634e2112e7 P 39e2d0baa860401bbf588f9f9e03fef2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf520b51778147d08f5fa929459dfd6e (127.22.104.129:34807), ff8bccc72e754c8cba77ead752f3fc4d (127.22.104.130:43443)
I20240502 20:45:30.710791 23700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f3469a24a147c2a9b0f2634e2112e7" candidate_uuid: "39e2d0baa860401bbf588f9f9e03fef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" is_pre_election: true
I20240502 20:45:30.710757 23624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f3469a24a147c2a9b0f2634e2112e7" candidate_uuid: "39e2d0baa860401bbf588f9f9e03fef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf520b51778147d08f5fa929459dfd6e" is_pre_election: true
I20240502 20:45:30.710963 23700 raft_consensus.cc:2379] T 49f3469a24a147c2a9b0f2634e2112e7 P ff8bccc72e754c8cba77ead752f3fc4d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39e2d0baa860401bbf588f9f9e03fef2 in current term 1: Already voted for candidate cf520b51778147d08f5fa929459dfd6e in this term.
I20240502 20:45:30.711258 23742 leader_election.cc:302] T 49f3469a24a147c2a9b0f2634e2112e7 P 39e2d0baa860401bbf588f9f9e03fef2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39e2d0baa860401bbf588f9f9e03fef2; no voters: cf520b51778147d08f5fa929459dfd6e, ff8bccc72e754c8cba77ead752f3fc4d
I20240502 20:45:30.711439 23831 raft_consensus.cc:3037] T 49f3469a24a147c2a9b0f2634e2112e7 P 39e2d0baa860401bbf588f9f9e03fef2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.712271 23831 raft_consensus.cc:2726] T 49f3469a24a147c2a9b0f2634e2112e7 P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 20:45:30.874503 23847 consensus_queue.cc:1035] T a37266ada21d4a17b45bdee1225c7e1a P ff8bccc72e754c8cba77ead752f3fc4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:30.877979 23847 consensus_queue.cc:1035] T a37266ada21d4a17b45bdee1225c7e1a P ff8bccc72e754c8cba77ead752f3fc4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:30.884495 23831 consensus_queue.cc:1035] T 0fc3ca8a4b274046b1c1842318c8db21 P 39e2d0baa860401bbf588f9f9e03fef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:30.887786 23831 consensus_queue.cc:1035] T 0fc3ca8a4b274046b1c1842318c8db21 P 39e2d0baa860401bbf588f9f9e03fef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:30.889592 23855 consensus_queue.cc:1035] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:30.892185 23855 consensus_queue.cc:1035] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:30.922250 23956 consensus_queue.cc:1035] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:30.926623 23956 consensus_queue.cc:1035] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:30.938198 23831 consensus_queue.cc:1035] T 6f7bab52adb145df84de04cb1b7ea315 P 39e2d0baa860401bbf588f9f9e03fef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:30.941313 23953 consensus_queue.cc:1035] T 6f7bab52adb145df84de04cb1b7ea315 P 39e2d0baa860401bbf588f9f9e03fef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:31.010823 23953 consensus_queue.cc:1035] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:31.013775 23831 consensus_queue.cc:1035] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:31.020838 23956 consensus_queue.cc:1035] T 74948efa3e364ee596ea1050cddaaae2 P cf520b51778147d08f5fa929459dfd6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:31.023646 23956 consensus_queue.cc:1035] T 74948efa3e364ee596ea1050cddaaae2 P cf520b51778147d08f5fa929459dfd6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:31.024122 23847 consensus_queue.cc:1035] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:31.026953 23847 consensus_queue.cc:1035] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:31.203445 23575 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:45:31.204778 23575 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 20:45:31.204927 23575 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:45:31.587777 23944 heartbeater.cc:503] Master 127.22.104.190:32947 was elected leader, sending a full tablet report...
I20240502 20:45:33.205257 23575 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:45:33.206104 23575 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 20:45:33.206178 23575 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:45:35.206471 23575 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:45:35.207300 23575 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 20:45:35.207374 23575 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:45:36.219609 23776 consensus_queue.cc:237] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } attrs { replace: true } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } } peers { permanent_uuid: "7e5c2bf991e945d7a845d292c7fc9b77" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 33565 } attrs { promote: true } }
I20240502 20:45:36.221575 23700 raft_consensus.cc:1261] T 13f3f643f725403eb35362f87f389145 P ff8bccc72e754c8cba77ead752f3fc4d [term 1 FOLLOWER]: Refusing update from remote peer 39e2d0baa860401bbf588f9f9e03fef2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:36.221662 23623 raft_consensus.cc:1261] T 13f3f643f725403eb35362f87f389145 P cf520b51778147d08f5fa929459dfd6e [term 1 FOLLOWER]: Refusing update from remote peer 39e2d0baa860401bbf588f9f9e03fef2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:36.222043 23953 consensus_queue.cc:1035] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:45:36.222606 23972 consensus_queue.cc:1035] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:45:36.224619 23973 raft_consensus.cc:2932] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e5c2bf991e945d7a845d292c7fc9b77 (127.22.104.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } attrs { replace: true } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } } peers { permanent_uuid: "7e5c2bf991e945d7a845d292c7fc9b77" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 33565 } attrs { promote: true } } }
I20240502 20:45:36.225157 23623 raft_consensus.cc:2932] T 13f3f643f725403eb35362f87f389145 P cf520b51778147d08f5fa929459dfd6e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e5c2bf991e945d7a845d292c7fc9b77 (127.22.104.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } attrs { replace: true } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } } peers { permanent_uuid: "7e5c2bf991e945d7a845d292c7fc9b77" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 33565 } attrs { promote: true } } }
W20240502 20:45:36.226478 23739 consensus_peers.cc:485] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2 -> Peer 7e5c2bf991e945d7a845d292c7fc9b77 (127.22.104.132:33565): Couldn't send request to peer 7e5c2bf991e945d7a845d292c7fc9b77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 13f3f643f725403eb35362f87f389145. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:36.227483 23517 catalog_manager.cc:5484] T 13f3f643f725403eb35362f87f389145 P cf520b51778147d08f5fa929459dfd6e reported cstate change: config changed from index -1 to 2, NON_VOTER 7e5c2bf991e945d7a845d292c7fc9b77 (127.22.104.132) added. New cstate: current_term: 1 leader_uuid: "39e2d0baa860401bbf588f9f9e03fef2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } attrs { replace: true } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } } peers { permanent_uuid: "7e5c2bf991e945d7a845d292c7fc9b77" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 33565 } attrs { promote: true } } }
I20240502 20:45:36.228216 23700 raft_consensus.cc:2932] T 13f3f643f725403eb35362f87f389145 P ff8bccc72e754c8cba77ead752f3fc4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e5c2bf991e945d7a845d292c7fc9b77 (127.22.104.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } attrs { replace: true } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } } peers { permanent_uuid: "7e5c2bf991e945d7a845d292c7fc9b77" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 33565 } attrs { promote: true } } }
I20240502 20:45:36.232524 23623 consensus_queue.cc:237] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } attrs { replace: true } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } } peers { permanent_uuid: "7e5c2bf991e945d7a845d292c7fc9b77" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 33565 } attrs { promote: true } }
I20240502 20:45:36.234099 23700 raft_consensus.cc:1261] T 1ee72890ba03492b9eebab393765739f P ff8bccc72e754c8cba77ead752f3fc4d [term 1 FOLLOWER]: Refusing update from remote peer cf520b51778147d08f5fa929459dfd6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:36.234164 23776 raft_consensus.cc:1261] T 1ee72890ba03492b9eebab393765739f P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 FOLLOWER]: Refusing update from remote peer cf520b51778147d08f5fa929459dfd6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:36.234450 23984 consensus_queue.cc:1035] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:45:36.234809 23985 consensus_queue.cc:1035] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:45:36.236583 23985 raft_consensus.cc:2932] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e5c2bf991e945d7a845d292c7fc9b77 (127.22.104.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } attrs { replace: true } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } } peers { permanent_uuid: "7e5c2bf991e945d7a845d292c7fc9b77" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 33565 } attrs { promote: true } } }
W20240502 20:45:36.237888 23587 consensus_peers.cc:485] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e -> Peer 7e5c2bf991e945d7a845d292c7fc9b77 (127.22.104.132:33565): Couldn't send request to peer 7e5c2bf991e945d7a845d292c7fc9b77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ee72890ba03492b9eebab393765739f. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:36.239145 23776 raft_consensus.cc:2932] T 1ee72890ba03492b9eebab393765739f P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e5c2bf991e945d7a845d292c7fc9b77 (127.22.104.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } attrs { replace: true } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } } peers { permanent_uuid: "7e5c2bf991e945d7a845d292c7fc9b77" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 33565 } attrs { promote: true } } }
I20240502 20:45:36.239575 23519 catalog_manager.cc:5484] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e reported cstate change: config changed from index -1 to 2, NON_VOTER 7e5c2bf991e945d7a845d292c7fc9b77 (127.22.104.132) added. New cstate: current_term: 1 leader_uuid: "cf520b51778147d08f5fa929459dfd6e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e5c2bf991e945d7a845d292c7fc9b77" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 33565 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:36.240567 23700 raft_consensus.cc:2932] T 1ee72890ba03492b9eebab393765739f P ff8bccc72e754c8cba77ead752f3fc4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e5c2bf991e945d7a845d292c7fc9b77 (127.22.104.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } attrs { replace: true } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } } peers { permanent_uuid: "7e5c2bf991e945d7a845d292c7fc9b77" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 33565 } attrs { promote: true } } }
I20240502 20:45:36.245961 23700 consensus_queue.cc:237] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } attrs { replace: true } } peers { permanent_uuid: "7e5c2bf991e945d7a845d292c7fc9b77" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 33565 } attrs { promote: true } }
I20240502 20:45:36.247331 23623 raft_consensus.cc:1261] T 1fa05319be264a8dacfba3fbb757629b P cf520b51778147d08f5fa929459dfd6e [term 1 FOLLOWER]: Refusing update from remote peer ff8bccc72e754c8cba77ead752f3fc4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:36.247383 23776 raft_consensus.cc:1261] T 1fa05319be264a8dacfba3fbb757629b P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 FOLLOWER]: Refusing update from remote peer ff8bccc72e754c8cba77ead752f3fc4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:36.247709 23991 consensus_queue.cc:1035] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:45:36.248049 23991 consensus_queue.cc:1035] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:45:36.250105 23991 raft_consensus.cc:2932] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e5c2bf991e945d7a845d292c7fc9b77 (127.22.104.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } attrs { replace: true } } peers { permanent_uuid: "7e5c2bf991e945d7a845d292c7fc9b77" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 33565 } attrs { promote: true } } }
W20240502 20:45:36.251686 23664 consensus_peers.cc:485] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d -> Peer 7e5c2bf991e945d7a845d292c7fc9b77 (127.22.104.132:33565): Couldn't send request to peer 7e5c2bf991e945d7a845d292c7fc9b77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1fa05319be264a8dacfba3fbb757629b. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:36.252769 23520 catalog_manager.cc:5484] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d reported cstate change: config changed from index -1 to 2, NON_VOTER 7e5c2bf991e945d7a845d292c7fc9b77 (127.22.104.132) added. New cstate: current_term: 1 leader_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e5c2bf991e945d7a845d292c7fc9b77" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 33565 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:36.253353 23623 consensus_queue.cc:237] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } attrs { replace: true } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } } peers { permanent_uuid: "7e5c2bf991e945d7a845d292c7fc9b77" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 33565 } attrs { promote: true } }
I20240502 20:45:36.253331 23624 raft_consensus.cc:2932] T 1fa05319be264a8dacfba3fbb757629b P cf520b51778147d08f5fa929459dfd6e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e5c2bf991e945d7a845d292c7fc9b77 (127.22.104.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } attrs { replace: true } } peers { permanent_uuid: "7e5c2bf991e945d7a845d292c7fc9b77" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 33565 } attrs { promote: true } } }
I20240502 20:45:36.254803 23700 raft_consensus.cc:1261] T 49f3469a24a147c2a9b0f2634e2112e7 P ff8bccc72e754c8cba77ead752f3fc4d [term 1 FOLLOWER]: Refusing update from remote peer cf520b51778147d08f5fa929459dfd6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:36.255373 23826 consensus_queue.cc:1035] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 20:45:36.255602 23587 consensus_peers.cc:485] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e -> Peer 7e5c2bf991e945d7a845d292c7fc9b77 (127.22.104.132:33565): Couldn't send request to peer 7e5c2bf991e945d7a845d292c7fc9b77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49f3469a24a147c2a9b0f2634e2112e7. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:36.255631 23775 raft_consensus.cc:2932] T 1fa05319be264a8dacfba3fbb757629b P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e5c2bf991e945d7a845d292c7fc9b77 (127.22.104.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } attrs { replace: true } } peers { permanent_uuid: "7e5c2bf991e945d7a845d292c7fc9b77" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 33565 } attrs { promote: true } } }
I20240502 20:45:36.256242 23776 raft_consensus.cc:1261] T 49f3469a24a147c2a9b0f2634e2112e7 P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 FOLLOWER]: Refusing update from remote peer cf520b51778147d08f5fa929459dfd6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:36.257512 23826 consensus_queue.cc:1035] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:45:36.259143 23826 raft_consensus.cc:2932] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e5c2bf991e945d7a845d292c7fc9b77 (127.22.104.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } attrs { replace: true } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } } peers { permanent_uuid: "7e5c2bf991e945d7a845d292c7fc9b77" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 33565 } attrs { promote: true } } }
I20240502 20:45:36.259511 23775 raft_consensus.cc:2932] T 49f3469a24a147c2a9b0f2634e2112e7 P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e5c2bf991e945d7a845d292c7fc9b77 (127.22.104.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } attrs { replace: true } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } } peers { permanent_uuid: "7e5c2bf991e945d7a845d292c7fc9b77" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 33565 } attrs { promote: true } } }
I20240502 20:45:36.261008 23700 raft_consensus.cc:2932] T 49f3469a24a147c2a9b0f2634e2112e7 P ff8bccc72e754c8cba77ead752f3fc4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e5c2bf991e945d7a845d292c7fc9b77 (127.22.104.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } attrs { replace: true } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } } peers { permanent_uuid: "7e5c2bf991e945d7a845d292c7fc9b77" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 33565 } attrs { promote: true } } }
I20240502 20:45:36.262229 23520 catalog_manager.cc:5484] T 49f3469a24a147c2a9b0f2634e2112e7 P 39e2d0baa860401bbf588f9f9e03fef2 reported cstate change: config changed from index -1 to 2, NON_VOTER 7e5c2bf991e945d7a845d292c7fc9b77 (127.22.104.132) added. New cstate: current_term: 1 leader_uuid: "cf520b51778147d08f5fa929459dfd6e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } attrs { replace: true } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } } peers { permanent_uuid: "7e5c2bf991e945d7a845d292c7fc9b77" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 33565 } attrs { promote: true } } }
I20240502 20:45:36.612771 22946 tablet_server.cc:178] TabletServer@127.22.104.129:0 shutting down...
I20240502 20:45:36.614266 24000 ts_tablet_manager.cc:902] T 13f3f643f725403eb35362f87f389145 P 7e5c2bf991e945d7a845d292c7fc9b77: Initiating tablet copy from peer 39e2d0baa860401bbf588f9f9e03fef2 (127.22.104.131:35163)
W20240502 20:45:36.615397 23507 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 1 similar messages]
I20240502 20:45:36.616034 24000 tablet_copy_client.cc:308] T 13f3f643f725403eb35362f87f389145 P 7e5c2bf991e945d7a845d292c7fc9b77: tablet copy: Beginning tablet copy session from remote peer at address 127.22.104.131:35163
W20240502 20:45:36.616298 23569 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 20:45:36.616362 23569 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 20:45:36.617902 22946 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 20:45:36.618188 22946 tablet_replica.cc:330] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e: stopping tablet replica
I20240502 20:45:36.618342 22946 raft_consensus.cc:2229] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e [term 1 LEADER]: Raft consensus shutting down.
W20240502 20:45:36.618554 23569 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 20:45:36.618597 22946 raft_consensus.cc:2258] T 1ee72890ba03492b9eebab393765739f P cf520b51778147d08f5fa929459dfd6e [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 20:45:36.618608 23569 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 20:45:36.619858 22946 tablet_replica.cc:330] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e: stopping tablet replica
I20240502 20:45:36.619968 22946 raft_consensus.cc:2229] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:45:36.620187 22946 raft_consensus.cc:2258] T 49f3469a24a147c2a9b0f2634e2112e7 P cf520b51778147d08f5fa929459dfd6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:36.620725 22946 tablet_replica.cc:330] T 74948efa3e364ee596ea1050cddaaae2 P cf520b51778147d08f5fa929459dfd6e: stopping tablet replica
I20240502 20:45:36.620834 22946 raft_consensus.cc:2229] T 74948efa3e364ee596ea1050cddaaae2 P cf520b51778147d08f5fa929459dfd6e [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:45:36.621031 22946 raft_consensus.cc:2258] T 74948efa3e364ee596ea1050cddaaae2 P cf520b51778147d08f5fa929459dfd6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:36.621049 23786 tablet_copy_service.cc:140] P 39e2d0baa860401bbf588f9f9e03fef2: Received BeginTabletCopySession request for tablet 13f3f643f725403eb35362f87f389145 from peer 7e5c2bf991e945d7a845d292c7fc9b77 ({username='slave'} at 127.0.0.1:59544)
I20240502 20:45:36.621160 23786 tablet_copy_service.cc:161] P 39e2d0baa860401bbf588f9f9e03fef2: Beginning new tablet copy session on tablet 13f3f643f725403eb35362f87f389145 from peer 7e5c2bf991e945d7a845d292c7fc9b77 at {username='slave'} at 127.0.0.1:59544: session id = 7e5c2bf991e945d7a845d292c7fc9b77-13f3f643f725403eb35362f87f389145
I20240502 20:45:36.621374 22946 tablet_replica.cc:330] T a37266ada21d4a17b45bdee1225c7e1a P cf520b51778147d08f5fa929459dfd6e: stopping tablet replica
I20240502 20:45:36.621471 22946 raft_consensus.cc:2229] T a37266ada21d4a17b45bdee1225c7e1a P cf520b51778147d08f5fa929459dfd6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:36.621551 22946 raft_consensus.cc:2258] T a37266ada21d4a17b45bdee1225c7e1a P cf520b51778147d08f5fa929459dfd6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:36.621865 22946 tablet_replica.cc:330] T 1fa05319be264a8dacfba3fbb757629b P cf520b51778147d08f5fa929459dfd6e: stopping tablet replica
I20240502 20:45:36.621943 22946 raft_consensus.cc:2229] T 1fa05319be264a8dacfba3fbb757629b P cf520b51778147d08f5fa929459dfd6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:36.622028 22946 raft_consensus.cc:2258] T 1fa05319be264a8dacfba3fbb757629b P cf520b51778147d08f5fa929459dfd6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:36.622459 22946 tablet_replica.cc:330] T 0fc3ca8a4b274046b1c1842318c8db21 P cf520b51778147d08f5fa929459dfd6e: stopping tablet replica
I20240502 20:45:36.622538 22946 raft_consensus.cc:2229] T 0fc3ca8a4b274046b1c1842318c8db21 P cf520b51778147d08f5fa929459dfd6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:36.622611 22946 raft_consensus.cc:2258] T 0fc3ca8a4b274046b1c1842318c8db21 P cf520b51778147d08f5fa929459dfd6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:36.622849 23786 tablet_copy_source_session.cc:205] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:36.622942 22946 tablet_replica.cc:330] T 6f7bab52adb145df84de04cb1b7ea315 P cf520b51778147d08f5fa929459dfd6e: stopping tablet replica
I20240502 20:45:36.623020 22946 raft_consensus.cc:2229] T 6f7bab52adb145df84de04cb1b7ea315 P cf520b51778147d08f5fa929459dfd6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:36.623111 22946 raft_consensus.cc:2258] T 6f7bab52adb145df84de04cb1b7ea315 P cf520b51778147d08f5fa929459dfd6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:36.623437 22946 tablet_replica.cc:330] T 13f3f643f725403eb35362f87f389145 P cf520b51778147d08f5fa929459dfd6e: stopping tablet replica
I20240502 20:45:36.623523 22946 raft_consensus.cc:2229] T 13f3f643f725403eb35362f87f389145 P cf520b51778147d08f5fa929459dfd6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:36.623610 22946 raft_consensus.cc:2258] T 13f3f643f725403eb35362f87f389145 P cf520b51778147d08f5fa929459dfd6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:36.624260 24000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13f3f643f725403eb35362f87f389145. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:36.626983 24000 tablet_copy_client.cc:783] T 13f3f643f725403eb35362f87f389145 P 7e5c2bf991e945d7a845d292c7fc9b77: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:36.627163 24000 tablet_copy_client.cc:647] T 13f3f643f725403eb35362f87f389145 P 7e5c2bf991e945d7a845d292c7fc9b77: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:36.628679 24000 tablet_copy_client.cc:523] T 13f3f643f725403eb35362f87f389145 P 7e5c2bf991e945d7a845d292c7fc9b77: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:36.629140 22946 tablet_server.cc:194] TabletServer@127.22.104.129:0 shutdown complete.
I20240502 20:45:36.630389 24000 tablet_bootstrap.cc:492] T 13f3f643f725403eb35362f87f389145 P 7e5c2bf991e945d7a845d292c7fc9b77: Bootstrap starting.
I20240502 20:45:36.631958 22946 tablet_server.cc:178] TabletServer@127.22.104.130:0 shutting down...
W20240502 20:45:36.634130 23569 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 20:45:36.634202 23569 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 20:45:36.636273 24000 tablet_bootstrap.cc:492] T 13f3f643f725403eb35362f87f389145 P 7e5c2bf991e945d7a845d292c7fc9b77: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:36.636603 24000 tablet_bootstrap.cc:492] T 13f3f643f725403eb35362f87f389145 P 7e5c2bf991e945d7a845d292c7fc9b77: Bootstrap complete.
I20240502 20:45:36.636822 24000 ts_tablet_manager.cc:1367] T 13f3f643f725403eb35362f87f389145 P 7e5c2bf991e945d7a845d292c7fc9b77: Time spent bootstrapping tablet: real 0.006s user 0.007s sys 0.000s
I20240502 20:45:36.637248 24000 raft_consensus.cc:348] T 13f3f643f725403eb35362f87f389145 P 7e5c2bf991e945d7a845d292c7fc9b77 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } attrs { replace: true } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } } peers { permanent_uuid: "7e5c2bf991e945d7a845d292c7fc9b77" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 33565 } attrs { promote: true } }
I20240502 20:45:36.637418 24000 raft_consensus.cc:729] T 13f3f643f725403eb35362f87f389145 P 7e5c2bf991e945d7a845d292c7fc9b77 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e5c2bf991e945d7a845d292c7fc9b77, State: Initialized, Role: LEARNER
I20240502 20:45:36.637583 24000 consensus_queue.cc:260] T 13f3f643f725403eb35362f87f389145 P 7e5c2bf991e945d7a845d292c7fc9b77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf520b51778147d08f5fa929459dfd6e" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 34807 } attrs { replace: true } } peers { permanent_uuid: "ff8bccc72e754c8cba77ead752f3fc4d" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 43443 } } peers { permanent_uuid: "39e2d0baa860401bbf588f9f9e03fef2" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 35163 } } peers { permanent_uuid: "7e5c2bf991e945d7a845d292c7fc9b77" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 33565 } attrs { promote: true } }
I20240502 20:45:36.638111 24000 ts_tablet_manager.cc:1398] T 13f3f643f725403eb35362f87f389145 P 7e5c2bf991e945d7a845d292c7fc9b77: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:36.638145 22946 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 20:45:36.638435 22946 tablet_replica.cc:330] T 1ee72890ba03492b9eebab393765739f P ff8bccc72e754c8cba77ead752f3fc4d: stopping tablet replica
I20240502 20:45:36.638553 22946 raft_consensus.cc:2229] T 1ee72890ba03492b9eebab393765739f P ff8bccc72e754c8cba77ead752f3fc4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:36.638656 23786 tablet_copy_service.cc:342] P 39e2d0baa860401bbf588f9f9e03fef2: Request end of tablet copy session 7e5c2bf991e945d7a845d292c7fc9b77-13f3f643f725403eb35362f87f389145 received from {username='slave'} at 127.0.0.1:59544
I20240502 20:45:36.638667 22946 raft_consensus.cc:2258] T 1ee72890ba03492b9eebab393765739f P ff8bccc72e754c8cba77ead752f3fc4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:36.638767 23786 tablet_copy_service.cc:433] P 39e2d0baa860401bbf588f9f9e03fef2: ending tablet copy session 7e5c2bf991e945d7a845d292c7fc9b77-13f3f643f725403eb35362f87f389145 on tablet 13f3f643f725403eb35362f87f389145 with peer 7e5c2bf991e945d7a845d292c7fc9b77
I20240502 20:45:36.639422 22946 tablet_replica.cc:330] T 74948efa3e364ee596ea1050cddaaae2 P ff8bccc72e754c8cba77ead752f3fc4d: stopping tablet replica
I20240502 20:45:36.639528 22946 raft_consensus.cc:2229] T 74948efa3e364ee596ea1050cddaaae2 P ff8bccc72e754c8cba77ead752f3fc4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:36.639623 22946 raft_consensus.cc:2258] T 74948efa3e364ee596ea1050cddaaae2 P ff8bccc72e754c8cba77ead752f3fc4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:36.639989 22946 tablet_replica.cc:330] T a37266ada21d4a17b45bdee1225c7e1a P ff8bccc72e754c8cba77ead752f3fc4d: stopping tablet replica
I20240502 20:45:36.640064 22946 raft_consensus.cc:2229] T a37266ada21d4a17b45bdee1225c7e1a P ff8bccc72e754c8cba77ead752f3fc4d [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:45:36.640209 22946 raft_consensus.cc:2258] T a37266ada21d4a17b45bdee1225c7e1a P ff8bccc72e754c8cba77ead752f3fc4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:36.640527 22946 tablet_replica.cc:330] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d: stopping tablet replica
I20240502 20:45:36.640621 22946 raft_consensus.cc:2229] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:45:36.640794 22946 raft_consensus.cc:2258] T 1fa05319be264a8dacfba3fbb757629b P ff8bccc72e754c8cba77ead752f3fc4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:36.641229 22946 tablet_replica.cc:330] T 49f3469a24a147c2a9b0f2634e2112e7 P ff8bccc72e754c8cba77ead752f3fc4d: stopping tablet replica
I20240502 20:45:36.641314 22946 raft_consensus.cc:2229] T 49f3469a24a147c2a9b0f2634e2112e7 P ff8bccc72e754c8cba77ead752f3fc4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:36.641414 22946 raft_consensus.cc:2258] T 49f3469a24a147c2a9b0f2634e2112e7 P ff8bccc72e754c8cba77ead752f3fc4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:36.641872 22946 tablet_replica.cc:330] T 13f3f643f725403eb35362f87f389145 P ff8bccc72e754c8cba77ead752f3fc4d: stopping tablet replica
I20240502 20:45:36.641964 22946 raft_consensus.cc:2229] T 13f3f643f725403eb35362f87f389145 P ff8bccc72e754c8cba77ead752f3fc4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:36.642060 22946 raft_consensus.cc:2258] T 13f3f643f725403eb35362f87f389145 P ff8bccc72e754c8cba77ead752f3fc4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:36.642491 22946 tablet_replica.cc:330] T 6f7bab52adb145df84de04cb1b7ea315 P ff8bccc72e754c8cba77ead752f3fc4d: stopping tablet replica
I20240502 20:45:36.642594 22946 raft_consensus.cc:2229] T 6f7bab52adb145df84de04cb1b7ea315 P ff8bccc72e754c8cba77ead752f3fc4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:36.642680 22946 raft_consensus.cc:2258] T 6f7bab52adb145df84de04cb1b7ea315 P ff8bccc72e754c8cba77ead752f3fc4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:36.642987 22946 tablet_replica.cc:330] T 0fc3ca8a4b274046b1c1842318c8db21 P ff8bccc72e754c8cba77ead752f3fc4d: stopping tablet replica
I20240502 20:45:36.643051 22946 raft_consensus.cc:2229] T 0fc3ca8a4b274046b1c1842318c8db21 P ff8bccc72e754c8cba77ead752f3fc4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:36.643131 22946 raft_consensus.cc:2258] T 0fc3ca8a4b274046b1c1842318c8db21 P ff8bccc72e754c8cba77ead752f3fc4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:36.646822 22946 tablet_server.cc:194] TabletServer@127.22.104.130:0 shutdown complete.
I20240502 20:45:36.649308 22946 tablet_server.cc:178] TabletServer@127.22.104.131:0 shutting down...
W20240502 20:45:36.650952 23569 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 20:45:36.651031 23569 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 20:45:36.654008 22946 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 20:45:36.654158 22946 tablet_replica.cc:330] T 1ee72890ba03492b9eebab393765739f P 39e2d0baa860401bbf588f9f9e03fef2: stopping tablet replica
I20240502 20:45:36.654284 22946 raft_consensus.cc:2229] T 1ee72890ba03492b9eebab393765739f P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:36.654394 22946 raft_consensus.cc:2258] T 1ee72890ba03492b9eebab393765739f P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:36.654979 22946 tablet_replica.cc:330] T 74948efa3e364ee596ea1050cddaaae2 P 39e2d0baa860401bbf588f9f9e03fef2: stopping tablet replica
I20240502 20:45:36.655068 22946 raft_consensus.cc:2229] T 74948efa3e364ee596ea1050cddaaae2 P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:36.655143 22946 raft_consensus.cc:2258] T 74948efa3e364ee596ea1050cddaaae2 P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:36.655454 22946 tablet_replica.cc:330] T 1fa05319be264a8dacfba3fbb757629b P 39e2d0baa860401bbf588f9f9e03fef2: stopping tablet replica
I20240502 20:45:36.655537 22946 raft_consensus.cc:2229] T 1fa05319be264a8dacfba3fbb757629b P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:36.655615 22946 raft_consensus.cc:2258] T 1fa05319be264a8dacfba3fbb757629b P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:36.656034 22946 tablet_replica.cc:330] T a37266ada21d4a17b45bdee1225c7e1a P 39e2d0baa860401bbf588f9f9e03fef2: stopping tablet replica
I20240502 20:45:36.656105 22946 raft_consensus.cc:2229] T a37266ada21d4a17b45bdee1225c7e1a P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:36.656172 22946 raft_consensus.cc:2258] T a37266ada21d4a17b45bdee1225c7e1a P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:36.656433 22946 tablet_replica.cc:330] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2: stopping tablet replica
I20240502 20:45:36.656530 22946 raft_consensus.cc:2229] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:45:36.656710 22946 raft_consensus.cc:2258] T 13f3f643f725403eb35362f87f389145 P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:36.657107 22946 tablet_replica.cc:330] T 6f7bab52adb145df84de04cb1b7ea315 P 39e2d0baa860401bbf588f9f9e03fef2: stopping tablet replica
I20240502 20:45:36.657176 22946 raft_consensus.cc:2229] T 6f7bab52adb145df84de04cb1b7ea315 P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:45:36.657305 22946 raft_consensus.cc:2258] T 6f7bab52adb145df84de04cb1b7ea315 P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:36.657560 22946 tablet_replica.cc:330] T 49f3469a24a147c2a9b0f2634e2112e7 P 39e2d0baa860401bbf588f9f9e03fef2: stopping tablet replica
I20240502 20:45:36.657622 22946 raft_consensus.cc:2229] T 49f3469a24a147c2a9b0f2634e2112e7 P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:36.657694 22946 raft_consensus.cc:2258] T 49f3469a24a147c2a9b0f2634e2112e7 P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:36.658056 22946 tablet_replica.cc:330] T 0fc3ca8a4b274046b1c1842318c8db21 P 39e2d0baa860401bbf588f9f9e03fef2: stopping tablet replica
I20240502 20:45:36.658119 22946 raft_consensus.cc:2229] T 0fc3ca8a4b274046b1c1842318c8db21 P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:45:36.658232 22946 raft_consensus.cc:2258] T 0fc3ca8a4b274046b1c1842318c8db21 P 39e2d0baa860401bbf588f9f9e03fef2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:36.671607 22946 tablet_server.cc:194] TabletServer@127.22.104.131:0 shutdown complete.
I20240502 20:45:36.674270 22946 tablet_server.cc:178] TabletServer@127.22.104.132:0 shutting down...
I20240502 20:45:36.677702 22946 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 20:45:36.677920 22946 tablet_replica.cc:330] T 13f3f643f725403eb35362f87f389145 P 7e5c2bf991e945d7a845d292c7fc9b77: stopping tablet replica
I20240502 20:45:36.678066 22946 raft_consensus.cc:2229] T 13f3f643f725403eb35362f87f389145 P 7e5c2bf991e945d7a845d292c7fc9b77 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 20:45:36.678181 22946 raft_consensus.cc:2258] T 13f3f643f725403eb35362f87f389145 P 7e5c2bf991e945d7a845d292c7fc9b77 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 20:45:36.691794 22946 tablet_server.cc:194] TabletServer@127.22.104.132:0 shutdown complete.
I20240502 20:45:36.693864 22946 master.cc:537] Master@127.22.104.190:32947 shutting down...
I20240502 20:45:36.697367 22946 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b73d6348bcd845d3a8febb5a143adea7 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:45:36.697516 22946 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b73d6348bcd845d3a8febb5a143adea7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:36.697567 22946 tablet_replica.cc:330] T 00000000000000000000000000000000 P b73d6348bcd845d3a8febb5a143adea7: stopping tablet replica
I20240502 20:45:36.710026 22946 master.cc:552] Master@127.22.104.190:32947 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (7542 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20240502 20:45:36.720225 22946 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.22.104.190:45795,127.22.104.189:44493,127.22.104.188:37977
I20240502 20:45:36.720602 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:36.722597 24010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:36.723098 24011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:36.723572 24013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:36.724004 22946 server_base.cc:1034] running on GCE node
I20240502 20:45:36.724174 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:45:36.724211 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:45:36.724231 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682736724231 us; error 0 us; skew 500 ppm
I20240502 20:45:36.725124 22946 webserver.cc:460] Webserver started at http://127.22.104.190:36117/ using document root <none> and password file <none>
I20240502 20:45:36.725281 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:36.725334 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:36.725411 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:36.725806 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714682726530369-22946-0/minicluster-data/master-0-root/instance:
uuid: "a0199d692a1540deb1233383a809f02b"
format_stamp: "Formatted at 2024-05-02 20:45:36 on dist-test-slave-rgns"
I20240502 20:45:36.727313 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 20:45:36.728227 24018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:36.728474 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 20:45:36.728542 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714682726530369-22946-0/minicluster-data/master-0-root
uuid: "a0199d692a1540deb1233383a809f02b"
format_stamp: "Formatted at 2024-05-02 20:45:36 on dist-test-slave-rgns"
I20240502 20:45:36.728610 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714682726530369-22946-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714682726530369-22946-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714682726530369-22946-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:36.746284 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:36.746683 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:36.750593 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.190:45795
I20240502 20:45:36.751847 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 20:45:36.756011 24073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.190:45795 every 8 connection(s)
I20240502 20:45:36.756477 24074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 20:45:36.757390 24076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:36.757596 24074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } has no permanent_uuid. Determining permanent_uuid...
W20240502 20:45:36.757854 24079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:36.758352 22946 server_base.cc:1034] running on GCE node
W20240502 20:45:36.758781 24077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:36.758953 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:45:36.758991 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:45:36.759011 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682736759011 us; error 0 us; skew 500 ppm
I20240502 20:45:36.759912 22946 webserver.cc:460] Webserver started at http://127.22.104.189:33061/ using document root <none> and password file <none>
I20240502 20:45:36.760047 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:36.760099 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:36.760172 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:36.760502 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714682726530369-22946-0/minicluster-data/master-1-root/instance:
uuid: "0ed633f4a06e4f85bdc8a892ba118320"
format_stamp: "Formatted at 2024-05-02 20:45:36 on dist-test-slave-rgns"
I20240502 20:45:36.762056 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 20:45:36.762197 24074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } has no permanent_uuid. Determining permanent_uuid...
I20240502 20:45:36.763031 24088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:36.763288 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:36.763355 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714682726530369-22946-0/minicluster-data/master-1-root
uuid: "0ed633f4a06e4f85bdc8a892ba118320"
format_stamp: "Formatted at 2024-05-02 20:45:36 on dist-test-slave-rgns"
I20240502 20:45:36.763423 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714682726530369-22946-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714682726530369-22946-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714682726530369-22946-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240502 20:45:36.763478 24074 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.104.189:44493: Network error: Client connection negotiation failed: client connection to 127.22.104.189:44493: connect: Connection refused (error 111)
I20240502 20:45:36.770445 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:36.770834 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:36.774555 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.189:44493
I20240502 20:45:36.774596 24141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.189:44493 every 8 connection(s)
I20240502 20:45:36.775240 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 20:45:36.775394 24142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:36.776767 24142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } has no permanent_uuid. Determining permanent_uuid...
W20240502 20:45:36.777148 24144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:36.777663 24145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:36.778126 24147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:36.778508 22946 server_base.cc:1034] running on GCE node
I20240502 20:45:36.778677 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:45:36.778753 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:45:36.778777 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682736778777 us; error 0 us; skew 500 ppm
I20240502 20:45:36.779716 22946 webserver.cc:460] Webserver started at http://127.22.104.188:46121/ using document root <none> and password file <none>
I20240502 20:45:36.779861 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:36.779917 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:36.779994 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:36.780380 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714682726530369-22946-0/minicluster-data/master-2-root/instance:
uuid: "75223d5a730946b68707abb230b536d0"
format_stamp: "Formatted at 2024-05-02 20:45:36 on dist-test-slave-rgns"
I20240502 20:45:36.781039 24142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } has no permanent_uuid. Determining permanent_uuid...
I20240502 20:45:36.781950 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 20:45:36.782924 24154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:36.783161 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:36.783218 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714682726530369-22946-0/minicluster-data/master-2-root
uuid: "75223d5a730946b68707abb230b536d0"
format_stamp: "Formatted at 2024-05-02 20:45:36 on dist-test-slave-rgns"
I20240502 20:45:36.783275 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714682726530369-22946-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714682726530369-22946-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714682726530369-22946-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:36.785058 24142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } has no permanent_uuid. Determining permanent_uuid...
W20240502 20:45:36.786128 24142 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.104.188:37977: Network error: Client connection negotiation failed: client connection to 127.22.104.188:37977: connect: Connection refused (error 111)
I20240502 20:45:36.798794 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:36.799154 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:36.799623 24074 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } attempt: 1
I20240502 20:45:36.803752 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.188:37977
I20240502 20:45:36.803784 24207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.188:37977 every 8 connection(s)
I20240502 20:45:36.803898 24074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } has no permanent_uuid. Determining permanent_uuid...
I20240502 20:45:36.804234 22946 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 20:45:36.804663 24208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:36.806001 24208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } has no permanent_uuid. Determining permanent_uuid...
I20240502 20:45:36.809737 24208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } has no permanent_uuid. Determining permanent_uuid...
I20240502 20:45:36.809845 24074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b: Bootstrap starting.
I20240502 20:45:36.810804 24074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:36.811743 24074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b: No bootstrap required, opened a new log
I20240502 20:45:36.812146 24074 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0199d692a1540deb1233383a809f02b" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } } peers { permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } } peers { permanent_uuid: "75223d5a730946b68707abb230b536d0" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } }
I20240502 20:45:36.812263 24074 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:36.812302 24074 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0199d692a1540deb1233383a809f02b, State: Initialized, Role: FOLLOWER
I20240502 20:45:36.812471 24074 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0199d692a1540deb1233383a809f02b" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } } peers { permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } } peers { permanent_uuid: "75223d5a730946b68707abb230b536d0" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } }
I20240502 20:45:36.812812 24213 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0199d692a1540deb1233383a809f02b" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } } peers { permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } } peers { permanent_uuid: "75223d5a730946b68707abb230b536d0" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } } }
I20240502 20:45:36.812922 24213 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b [sys.catalog]: This master's current role is: FOLLOWER
I20240502 20:45:36.812994 24074 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b [sys.catalog]: configured and running, proceeding with master startup.
I20240502 20:45:36.813148 24208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } has no permanent_uuid. Determining permanent_uuid...
I20240502 20:45:36.813258 24142 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } attempt: 1
I20240502 20:45:36.814792 22946 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20240502 20:45:36.815274 24225 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 20:45:36.815330 24225 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 20:45:36.817737 24208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0: Bootstrap starting.
I20240502 20:45:36.817826 24142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320: Bootstrap starting.
I20240502 20:45:36.818420 24208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:36.818486 24142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:36.819355 24208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0: No bootstrap required, opened a new log
I20240502 20:45:36.819380 24142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320: No bootstrap required, opened a new log
I20240502 20:45:36.819730 24142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0199d692a1540deb1233383a809f02b" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } } peers { permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } } peers { permanent_uuid: "75223d5a730946b68707abb230b536d0" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } }
I20240502 20:45:36.819741 24208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0199d692a1540deb1233383a809f02b" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } } peers { permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } } peers { permanent_uuid: "75223d5a730946b68707abb230b536d0" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } }
I20240502 20:45:36.819818 24142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:36.819834 24208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:36.819839 24142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed633f4a06e4f85bdc8a892ba118320, State: Initialized, Role: FOLLOWER
I20240502 20:45:36.819883 24208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75223d5a730946b68707abb230b536d0, State: Initialized, Role: FOLLOWER
I20240502 20:45:36.819983 24142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0199d692a1540deb1233383a809f02b" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } } peers { permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } } peers { permanent_uuid: "75223d5a730946b68707abb230b536d0" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } }
I20240502 20:45:36.820001 24208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0199d692a1540deb1233383a809f02b" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } } peers { permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } } peers { permanent_uuid: "75223d5a730946b68707abb230b536d0" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } }
I20240502 20:45:36.820250 24228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0199d692a1540deb1233383a809f02b" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } } peers { permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } } peers { permanent_uuid: "75223d5a730946b68707abb230b536d0" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } } }
I20240502 20:45:36.820288 24229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0199d692a1540deb1233383a809f02b" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } } peers { permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } } peers { permanent_uuid: "75223d5a730946b68707abb230b536d0" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } } }
I20240502 20:45:36.820438 24208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 20:45:36.820431 24142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 20:45:36.820495 24229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 20:45:36.820402 24228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 20:45:36.822103 22946 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240502 20:45:36.822543 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:36.822664 24251 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 20:45:36.822758 24251 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 20:45:36.822664 24250 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 20:45:36.822825 24250 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 20:45:36.824545 24253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:36.824960 24252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:36.825654 24255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:36.826170 22946 server_base.cc:1034] running on GCE node
I20240502 20:45:36.826308 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:45:36.826344 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:45:36.826364 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682736826363 us; error 0 us; skew 500 ppm
I20240502 20:45:36.827284 22946 webserver.cc:460] Webserver started at http://127.22.104.129:41059/ using document root <none> and password file <none>
I20240502 20:45:36.827415 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:36.827457 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:36.827525 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:36.827888 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714682726530369-22946-0/minicluster-data/ts-0-root/instance:
uuid: "64542b4dcd724f9cad68e8ba06e2b1c1"
format_stamp: "Formatted at 2024-05-02 20:45:36 on dist-test-slave-rgns"
I20240502 20:45:36.829162 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:36.829950 24260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:36.830161 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:36.830224 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714682726530369-22946-0/minicluster-data/ts-0-root
uuid: "64542b4dcd724f9cad68e8ba06e2b1c1"
format_stamp: "Formatted at 2024-05-02 20:45:36 on dist-test-slave-rgns"
I20240502 20:45:36.830282 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714682726530369-22946-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714682726530369-22946-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714682726530369-22946-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:36.836511 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:36.836831 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:36.837080 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:36.837502 22946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:36.837538 22946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:36.837575 22946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:36.837608 22946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:36.841866 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.129:41685
I20240502 20:45:36.841919 24323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.129:41685 every 8 connection(s)
I20240502 20:45:36.842808 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:36.845855 24332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:36.846241 24331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:36.846711 24334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:36.847030 24325 heartbeater.cc:347] Connected to a master server at 127.22.104.190:45795
I20240502 20:45:36.847134 24325 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:36.847304 22946 server_base.cc:1034] running on GCE node
I20240502 20:45:36.847458 24325 heartbeater.cc:511] Master 127.22.104.190:45795 requested a full tablet report, sending...
I20240502 20:45:36.847487 24326 heartbeater.cc:347] Connected to a master server at 127.22.104.189:44493
I20240502 20:45:36.847494 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240502 20:45:36.847556 24326 heartbeater.cc:464] Registering TS with master...
W20240502 20:45:36.847572 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:45:36.847594 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682736847594 us; error 0 us; skew 500 ppm
I20240502 20:45:36.847746 24326 heartbeater.cc:511] Master 127.22.104.189:44493 requested a full tablet report, sending...
I20240502 20:45:36.847870 24324 heartbeater.cc:347] Connected to a master server at 127.22.104.188:37977
I20240502 20:45:36.847949 24324 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:36.848150 24324 heartbeater.cc:511] Master 127.22.104.188:37977 requested a full tablet report, sending...
I20240502 20:45:36.848219 24036 ts_manager.cc:194] Registered new tserver with Master: 64542b4dcd724f9cad68e8ba06e2b1c1 (127.22.104.129:41685)
I20240502 20:45:36.848629 22946 webserver.cc:460] Webserver started at http://127.22.104.130:40011/ using document root <none> and password file <none>
I20240502 20:45:36.848750 24172 ts_manager.cc:194] Registered new tserver with Master: 64542b4dcd724f9cad68e8ba06e2b1c1 (127.22.104.129:41685)
I20240502 20:45:36.848781 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:36.848835 24105 ts_manager.cc:194] Registered new tserver with Master: 64542b4dcd724f9cad68e8ba06e2b1c1 (127.22.104.129:41685)
I20240502 20:45:36.848857 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:36.848950 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:36.849361 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714682726530369-22946-0/minicluster-data/ts-1-root/instance:
uuid: "4cc73560bb1d44959f11d6b229819f83"
format_stamp: "Formatted at 2024-05-02 20:45:36 on dist-test-slave-rgns"
I20240502 20:45:36.850642 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:36.851480 24339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:36.851687 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:36.851750 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714682726530369-22946-0/minicluster-data/ts-1-root
uuid: "4cc73560bb1d44959f11d6b229819f83"
format_stamp: "Formatted at 2024-05-02 20:45:36 on dist-test-slave-rgns"
I20240502 20:45:36.851809 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714682726530369-22946-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714682726530369-22946-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714682726530369-22946-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:36.863492 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:36.863894 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:36.864197 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:36.864670 22946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:36.864710 22946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:36.864751 22946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:36.864778 22946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:36.869066 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.130:44453
I20240502 20:45:36.869145 24403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.130:44453 every 8 connection(s)
I20240502 20:45:36.870003 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:36.873102 24412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:36.873521 24411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:36.873926 24415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:36.874368 22946 server_base.cc:1034] running on GCE node
I20240502 20:45:36.874580 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240502 20:45:36.874665 24405 heartbeater.cc:347] Connected to a master server at 127.22.104.190:45795
I20240502 20:45:36.874780 24405 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:36.874784 24406 heartbeater.cc:347] Connected to a master server at 127.22.104.189:44493
I20240502 20:45:36.874881 24406 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:36.874955 24404 heartbeater.cc:347] Connected to a master server at 127.22.104.188:37977
I20240502 20:45:36.874990 24405 heartbeater.cc:511] Master 127.22.104.190:45795 requested a full tablet report, sending...
I20240502 20:45:36.875008 24404 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:36.875077 24406 heartbeater.cc:511] Master 127.22.104.189:44493 requested a full tablet report, sending...
I20240502 20:45:36.875154 24404 heartbeater.cc:511] Master 127.22.104.188:37977 requested a full tablet report, sending...
W20240502 20:45:36.874672 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:45:36.875280 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682736875279 us; error 0 us; skew 500 ppm
I20240502 20:45:36.875689 24105 ts_manager.cc:194] Registered new tserver with Master: 4cc73560bb1d44959f11d6b229819f83 (127.22.104.130:44453)
I20240502 20:45:36.875689 24036 ts_manager.cc:194] Registered new tserver with Master: 4cc73560bb1d44959f11d6b229819f83 (127.22.104.130:44453)
I20240502 20:45:36.876263 24172 ts_manager.cc:194] Registered new tserver with Master: 4cc73560bb1d44959f11d6b229819f83 (127.22.104.130:44453)
I20240502 20:45:36.876514 22946 webserver.cc:460] Webserver started at http://127.22.104.131:40553/ using document root <none> and password file <none>
I20240502 20:45:36.876704 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:36.876749 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:36.876806 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:36.877161 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714682726530369-22946-0/minicluster-data/ts-2-root/instance:
uuid: "aa5824746b5d46319a368bf4e861c19e"
format_stamp: "Formatted at 2024-05-02 20:45:36 on dist-test-slave-rgns"
I20240502 20:45:36.878481 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:36.879346 24420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:36.879557 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:36.879621 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714682726530369-22946-0/minicluster-data/ts-2-root
uuid: "aa5824746b5d46319a368bf4e861c19e"
format_stamp: "Formatted at 2024-05-02 20:45:36 on dist-test-slave-rgns"
I20240502 20:45:36.879684 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714682726530369-22946-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714682726530369-22946-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714682726530369-22946-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:36.883980 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:36.884290 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:36.884523 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:36.884909 22946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:36.884944 22946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:36.884974 22946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:36.885000 22946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:36.889286 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.131:39511
I20240502 20:45:36.889393 24483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.131:39511 every 8 connection(s)
I20240502 20:45:36.894554 24484 heartbeater.cc:347] Connected to a master server at 127.22.104.188:37977
I20240502 20:45:36.894604 24485 heartbeater.cc:347] Connected to a master server at 127.22.104.190:45795
I20240502 20:45:36.894713 24486 heartbeater.cc:347] Connected to a master server at 127.22.104.189:44493
I20240502 20:45:36.894758 24485 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:36.894739 24484 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:36.894812 24486 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:36.895023 24486 heartbeater.cc:511] Master 127.22.104.189:44493 requested a full tablet report, sending...
I20240502 20:45:36.895010 24484 heartbeater.cc:511] Master 127.22.104.188:37977 requested a full tablet report, sending...
I20240502 20:45:36.895010 24485 heartbeater.cc:511] Master 127.22.104.190:45795 requested a full tablet report, sending...
I20240502 20:45:36.895702 24105 ts_manager.cc:194] Registered new tserver with Master: aa5824746b5d46319a368bf4e861c19e (127.22.104.131:39511)
I20240502 20:45:36.895695 24172 ts_manager.cc:194] Registered new tserver with Master: aa5824746b5d46319a368bf4e861c19e (127.22.104.131:39511)
I20240502 20:45:36.895695 24036 ts_manager.cc:194] Registered new tserver with Master: aa5824746b5d46319a368bf4e861c19e (127.22.104.131:39511)
I20240502 20:45:36.895884 22946 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006101598s
I20240502 20:45:37.051605 24213 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:37.051769 24213 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0199d692a1540deb1233383a809f02b" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } } peers { permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } } peers { permanent_uuid: "75223d5a730946b68707abb230b536d0" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } }
I20240502 20:45:37.052254 24213 leader_election.cc:288] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ed633f4a06e4f85bdc8a892ba118320 (127.22.104.189:44493), 75223d5a730946b68707abb230b536d0 (127.22.104.188:37977)
I20240502 20:45:37.052701 24182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a0199d692a1540deb1233383a809f02b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75223d5a730946b68707abb230b536d0" is_pre_election: true
I20240502 20:45:37.052704 24116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a0199d692a1540deb1233383a809f02b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed633f4a06e4f85bdc8a892ba118320" is_pre_election: true
I20240502 20:45:37.052865 24182 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0199d692a1540deb1233383a809f02b in term 0.
I20240502 20:45:37.052878 24116 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0199d692a1540deb1233383a809f02b in term 0.
I20240502 20:45:37.053212 24022 leader_election.cc:302] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed633f4a06e4f85bdc8a892ba118320, a0199d692a1540deb1233383a809f02b; no voters:
I20240502 20:45:37.053381 24213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:37.053443 24213 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:37.053469 24213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:37.054286 24213 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0199d692a1540deb1233383a809f02b" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } } peers { permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } } peers { permanent_uuid: "75223d5a730946b68707abb230b536d0" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } }
I20240502 20:45:37.054610 24213 leader_election.cc:288] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b [CANDIDATE]: Term 1 election: Requested vote from peers 0ed633f4a06e4f85bdc8a892ba118320 (127.22.104.189:44493), 75223d5a730946b68707abb230b536d0 (127.22.104.188:37977)
I20240502 20:45:37.054935 24116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a0199d692a1540deb1233383a809f02b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed633f4a06e4f85bdc8a892ba118320"
I20240502 20:45:37.054946 24182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a0199d692a1540deb1233383a809f02b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75223d5a730946b68707abb230b536d0"
I20240502 20:45:37.055078 24116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:37.055105 24182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:37.056046 24182 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0199d692a1540deb1233383a809f02b in term 1.
I20240502 20:45:37.056046 24116 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0199d692a1540deb1233383a809f02b in term 1.
I20240502 20:45:37.056350 24021 leader_election.cc:302] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75223d5a730946b68707abb230b536d0, a0199d692a1540deb1233383a809f02b; no voters:
I20240502 20:45:37.056497 24213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:37.056705 24213 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b [term 1 LEADER]: Becoming Leader. State: Replica: a0199d692a1540deb1233383a809f02b, State: Running, Role: LEADER
I20240502 20:45:37.056851 24213 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0199d692a1540deb1233383a809f02b" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } } peers { permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } } peers { permanent_uuid: "75223d5a730946b68707abb230b536d0" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } }
I20240502 20:45:37.057469 24492 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0199d692a1540deb1233383a809f02b. Latest consensus state: current_term: 1 leader_uuid: "a0199d692a1540deb1233383a809f02b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0199d692a1540deb1233383a809f02b" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } } peers { permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } } peers { permanent_uuid: "75223d5a730946b68707abb230b536d0" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } } }
I20240502 20:45:37.057575 24492 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:37.057861 24494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 20:45:37.058591 24494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 20:45:37.060395 24182 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [term 1 FOLLOWER]: Refusing update from remote peer a0199d692a1540deb1233383a809f02b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:37.060395 24116 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [term 1 FOLLOWER]: Refusing update from remote peer a0199d692a1540deb1233383a809f02b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:37.060846 24492 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b [LEADER]: Connected to new peer: Peer: permanent_uuid: "75223d5a730946b68707abb230b536d0" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:37.061003 24213 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:37.062230 24229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0199d692a1540deb1233383a809f02b. Latest consensus state: current_term: 1 leader_uuid: "a0199d692a1540deb1233383a809f02b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0199d692a1540deb1233383a809f02b" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } } peers { permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } } peers { permanent_uuid: "75223d5a730946b68707abb230b536d0" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } } }
I20240502 20:45:37.062348 24229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 20:45:37.062244 24228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0199d692a1540deb1233383a809f02b. Latest consensus state: current_term: 1 leader_uuid: "a0199d692a1540deb1233383a809f02b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0199d692a1540deb1233383a809f02b" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } } peers { permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } } peers { permanent_uuid: "75223d5a730946b68707abb230b536d0" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } } }
I20240502 20:45:37.062471 24228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 20:45:37.063460 24492 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a0199d692a1540deb1233383a809f02b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0199d692a1540deb1233383a809f02b" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } } peers { permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } } peers { permanent_uuid: "75223d5a730946b68707abb230b536d0" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } } }
I20240502 20:45:37.063570 24492 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:37.063694 24500 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a0199d692a1540deb1233383a809f02b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0199d692a1540deb1233383a809f02b" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } } peers { permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } } peers { permanent_uuid: "75223d5a730946b68707abb230b536d0" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } } }
I20240502 20:45:37.063764 24500 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:37.063905 24228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a0199d692a1540deb1233383a809f02b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0199d692a1540deb1233383a809f02b" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } } peers { permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } } peers { permanent_uuid: "75223d5a730946b68707abb230b536d0" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } } }
I20240502 20:45:37.063994 24228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 20:45:37.064195 24229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a0199d692a1540deb1233383a809f02b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0199d692a1540deb1233383a809f02b" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } } peers { permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } } peers { permanent_uuid: "75223d5a730946b68707abb230b536d0" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } } }
I20240502 20:45:37.064286 24229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 20:45:37.064725 24494 catalog_manager.cc:1347] Generated new cluster ID: 7dce8b09fcdd4dd496c043b6709256ff
I20240502 20:45:37.064795 24494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 20:45:37.074060 24494 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 20:45:37.074813 24494 catalog_manager.cc:1504] Loading token signing keys...
I20240502 20:45:37.081398 24494 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b: Generated new TSK 0
I20240502 20:45:37.081560 24494 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 20:45:37.823820 24251 catalog_manager.cc:1259] Loaded cluster ID: 7dce8b09fcdd4dd496c043b6709256ff
I20240502 20:45:37.823820 24250 catalog_manager.cc:1259] Loaded cluster ID: 7dce8b09fcdd4dd496c043b6709256ff
I20240502 20:45:37.823951 24251 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0: loading cluster ID for follower catalog manager: success
I20240502 20:45:37.824007 24250 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320: loading cluster ID for follower catalog manager: success
I20240502 20:45:37.825204 24251 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0: acquiring CA information for follower catalog manager: success
I20240502 20:45:37.825227 24250 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320: acquiring CA information for follower catalog manager: success
I20240502 20:45:37.825706 24250 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 20:45:37.825724 24251 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 20:45:37.851001 24036 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56152
I20240502 20:45:37.877858 24036 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56164
I20240502 20:45:37.898478 24036 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56178
I20240502 20:45:38.814968 24224 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:45:38.851891 24325 heartbeater.cc:503] Master 127.22.104.190:45795 was elected leader, sending a full tablet report...
I20240502 20:45:38.878789 24405 heartbeater.cc:503] Master 127.22.104.190:45795 was elected leader, sending a full tablet report...
I20240502 20:45:38.899358 24485 heartbeater.cc:503] Master 127.22.104.190:45795 was elected leader, sending a full tablet report...
I20240502 20:45:39.152096 22946 test_util.cc:268] Using random seed: -1349397285
I20240502 20:45:39.161164 24036 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56186:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 20:45:39.174752 24288 tablet_service.cc:1467] Processing CreateTablet for tablet 9bcbaf8c66744e948862d0adcce47d2a (DEFAULT_TABLE table=test-workload [id=f273f31e38fb44d5b24a754fae79d491]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240502 20:45:39.175079 24288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bcbaf8c66744e948862d0adcce47d2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:39.176877 24448 tablet_service.cc:1467] Processing CreateTablet for tablet b3ffa316dc07401e8ffc1a1a521a5ed3 (DEFAULT_TABLE table=test-workload [id=f273f31e38fb44d5b24a754fae79d491]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240502 20:45:39.177143 24448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3ffa316dc07401e8ffc1a1a521a5ed3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:39.177155 24533 tablet_bootstrap.cc:492] T 9bcbaf8c66744e948862d0adcce47d2a P 64542b4dcd724f9cad68e8ba06e2b1c1: Bootstrap starting.
I20240502 20:45:39.177421 24367 tablet_service.cc:1467] Processing CreateTablet for tablet ad2c408894064c3cbec0c88850ff85b4 (DEFAULT_TABLE table=test-workload [id=f273f31e38fb44d5b24a754fae79d491]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240502 20:45:39.177675 24367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad2c408894064c3cbec0c88850ff85b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:39.178145 24533 tablet_bootstrap.cc:654] T 9bcbaf8c66744e948862d0adcce47d2a P 64542b4dcd724f9cad68e8ba06e2b1c1: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:39.179333 24533 tablet_bootstrap.cc:492] T 9bcbaf8c66744e948862d0adcce47d2a P 64542b4dcd724f9cad68e8ba06e2b1c1: No bootstrap required, opened a new log
I20240502 20:45:39.179338 24535 tablet_bootstrap.cc:492] T b3ffa316dc07401e8ffc1a1a521a5ed3 P aa5824746b5d46319a368bf4e861c19e: Bootstrap starting.
I20240502 20:45:39.179433 24533 ts_tablet_manager.cc:1367] T 9bcbaf8c66744e948862d0adcce47d2a P 64542b4dcd724f9cad68e8ba06e2b1c1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 20:45:39.180002 24533 raft_consensus.cc:348] T 9bcbaf8c66744e948862d0adcce47d2a P 64542b4dcd724f9cad68e8ba06e2b1c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64542b4dcd724f9cad68e8ba06e2b1c1" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41685 } }
I20240502 20:45:39.180099 24533 raft_consensus.cc:374] T 9bcbaf8c66744e948862d0adcce47d2a P 64542b4dcd724f9cad68e8ba06e2b1c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:39.180136 24533 raft_consensus.cc:729] T 9bcbaf8c66744e948862d0adcce47d2a P 64542b4dcd724f9cad68e8ba06e2b1c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64542b4dcd724f9cad68e8ba06e2b1c1, State: Initialized, Role: FOLLOWER
I20240502 20:45:39.180270 24533 consensus_queue.cc:260] T 9bcbaf8c66744e948862d0adcce47d2a P 64542b4dcd724f9cad68e8ba06e2b1c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64542b4dcd724f9cad68e8ba06e2b1c1" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41685 } }
I20240502 20:45:39.180338 24536 tablet_bootstrap.cc:492] T ad2c408894064c3cbec0c88850ff85b4 P 4cc73560bb1d44959f11d6b229819f83: Bootstrap starting.
I20240502 20:45:39.180346 24533 raft_consensus.cc:388] T 9bcbaf8c66744e948862d0adcce47d2a P 64542b4dcd724f9cad68e8ba06e2b1c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:39.180431 24533 raft_consensus.cc:482] T 9bcbaf8c66744e948862d0adcce47d2a P 64542b4dcd724f9cad68e8ba06e2b1c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:39.180466 24535 tablet_bootstrap.cc:654] T b3ffa316dc07401e8ffc1a1a521a5ed3 P aa5824746b5d46319a368bf4e861c19e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:39.180490 24533 raft_consensus.cc:3037] T 9bcbaf8c66744e948862d0adcce47d2a P 64542b4dcd724f9cad68e8ba06e2b1c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:39.181192 24536 tablet_bootstrap.cc:654] T ad2c408894064c3cbec0c88850ff85b4 P 4cc73560bb1d44959f11d6b229819f83: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:39.181244 24533 raft_consensus.cc:504] T 9bcbaf8c66744e948862d0adcce47d2a P 64542b4dcd724f9cad68e8ba06e2b1c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64542b4dcd724f9cad68e8ba06e2b1c1" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41685 } }
I20240502 20:45:39.181380 24533 leader_election.cc:302] T 9bcbaf8c66744e948862d0adcce47d2a P 64542b4dcd724f9cad68e8ba06e2b1c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64542b4dcd724f9cad68e8ba06e2b1c1; no voters:
I20240502 20:45:39.181483 24535 tablet_bootstrap.cc:492] T b3ffa316dc07401e8ffc1a1a521a5ed3 P aa5824746b5d46319a368bf4e861c19e: No bootstrap required, opened a new log
I20240502 20:45:39.181557 24533 leader_election.cc:288] T 9bcbaf8c66744e948862d0adcce47d2a P 64542b4dcd724f9cad68e8ba06e2b1c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:39.181573 24535 ts_tablet_manager.cc:1367] T b3ffa316dc07401e8ffc1a1a521a5ed3 P aa5824746b5d46319a368bf4e861c19e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:39.181831 24540 raft_consensus.cc:2781] T 9bcbaf8c66744e948862d0adcce47d2a P 64542b4dcd724f9cad68e8ba06e2b1c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:39.181918 24533 ts_tablet_manager.cc:1398] T 9bcbaf8c66744e948862d0adcce47d2a P 64542b4dcd724f9cad68e8ba06e2b1c1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 20:45:39.182057 24535 raft_consensus.cc:348] T b3ffa316dc07401e8ffc1a1a521a5ed3 P aa5824746b5d46319a368bf4e861c19e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa5824746b5d46319a368bf4e861c19e" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 39511 } }
I20240502 20:45:39.182144 24535 raft_consensus.cc:374] T b3ffa316dc07401e8ffc1a1a521a5ed3 P aa5824746b5d46319a368bf4e861c19e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:39.182176 24535 raft_consensus.cc:729] T b3ffa316dc07401e8ffc1a1a521a5ed3 P aa5824746b5d46319a368bf4e861c19e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa5824746b5d46319a368bf4e861c19e, State: Initialized, Role: FOLLOWER
I20240502 20:45:39.182267 24536 tablet_bootstrap.cc:492] T ad2c408894064c3cbec0c88850ff85b4 P 4cc73560bb1d44959f11d6b229819f83: No bootstrap required, opened a new log
I20240502 20:45:39.182307 24535 consensus_queue.cc:260] T b3ffa316dc07401e8ffc1a1a521a5ed3 P aa5824746b5d46319a368bf4e861c19e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa5824746b5d46319a368bf4e861c19e" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 39511 } }
I20240502 20:45:39.182346 24536 ts_tablet_manager.cc:1367] T ad2c408894064c3cbec0c88850ff85b4 P 4cc73560bb1d44959f11d6b229819f83: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:39.182405 24535 raft_consensus.cc:388] T b3ffa316dc07401e8ffc1a1a521a5ed3 P aa5824746b5d46319a368bf4e861c19e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:39.182448 24535 raft_consensus.cc:482] T b3ffa316dc07401e8ffc1a1a521a5ed3 P aa5824746b5d46319a368bf4e861c19e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:39.182498 24535 raft_consensus.cc:3037] T b3ffa316dc07401e8ffc1a1a521a5ed3 P aa5824746b5d46319a368bf4e861c19e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:39.183123 24536 raft_consensus.cc:348] T ad2c408894064c3cbec0c88850ff85b4 P 4cc73560bb1d44959f11d6b229819f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc73560bb1d44959f11d6b229819f83" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 44453 } }
I20240502 20:45:39.183223 24536 raft_consensus.cc:374] T ad2c408894064c3cbec0c88850ff85b4 P 4cc73560bb1d44959f11d6b229819f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:39.183266 24536 raft_consensus.cc:729] T ad2c408894064c3cbec0c88850ff85b4 P 4cc73560bb1d44959f11d6b229819f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc73560bb1d44959f11d6b229819f83, State: Initialized, Role: FOLLOWER
I20240502 20:45:39.183384 24535 raft_consensus.cc:504] T b3ffa316dc07401e8ffc1a1a521a5ed3 P aa5824746b5d46319a368bf4e861c19e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa5824746b5d46319a368bf4e861c19e" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 39511 } }
I20240502 20:45:39.183391 24536 consensus_queue.cc:260] T ad2c408894064c3cbec0c88850ff85b4 P 4cc73560bb1d44959f11d6b229819f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc73560bb1d44959f11d6b229819f83" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 44453 } }
I20240502 20:45:39.183486 24540 raft_consensus.cc:686] T 9bcbaf8c66744e948862d0adcce47d2a P 64542b4dcd724f9cad68e8ba06e2b1c1 [term 1 LEADER]: Becoming Leader. State: Replica: 64542b4dcd724f9cad68e8ba06e2b1c1, State: Running, Role: LEADER
I20240502 20:45:39.183526 24535 leader_election.cc:302] T b3ffa316dc07401e8ffc1a1a521a5ed3 P aa5824746b5d46319a368bf4e861c19e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa5824746b5d46319a368bf4e861c19e; no voters:
I20240502 20:45:39.183557 24536 raft_consensus.cc:388] T ad2c408894064c3cbec0c88850ff85b4 P 4cc73560bb1d44959f11d6b229819f83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:39.183604 24536 raft_consensus.cc:482] T ad2c408894064c3cbec0c88850ff85b4 P 4cc73560bb1d44959f11d6b229819f83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:39.183651 24536 raft_consensus.cc:3037] T ad2c408894064c3cbec0c88850ff85b4 P 4cc73560bb1d44959f11d6b229819f83 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:39.183621 24540 consensus_queue.cc:237] T 9bcbaf8c66744e948862d0adcce47d2a P 64542b4dcd724f9cad68e8ba06e2b1c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64542b4dcd724f9cad68e8ba06e2b1c1" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41685 } }
I20240502 20:45:39.183697 24535 leader_election.cc:288] T b3ffa316dc07401e8ffc1a1a521a5ed3 P aa5824746b5d46319a368bf4e861c19e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:39.183847 24542 raft_consensus.cc:2781] T b3ffa316dc07401e8ffc1a1a521a5ed3 P aa5824746b5d46319a368bf4e861c19e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:39.183969 24535 ts_tablet_manager.cc:1398] T b3ffa316dc07401e8ffc1a1a521a5ed3 P aa5824746b5d46319a368bf4e861c19e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:39.184176 24542 raft_consensus.cc:686] T b3ffa316dc07401e8ffc1a1a521a5ed3 P aa5824746b5d46319a368bf4e861c19e [term 1 LEADER]: Becoming Leader. State: Replica: aa5824746b5d46319a368bf4e861c19e, State: Running, Role: LEADER
I20240502 20:45:39.184340 24542 consensus_queue.cc:237] T b3ffa316dc07401e8ffc1a1a521a5ed3 P aa5824746b5d46319a368bf4e861c19e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa5824746b5d46319a368bf4e861c19e" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 39511 } }
I20240502 20:45:39.184604 24536 raft_consensus.cc:504] T ad2c408894064c3cbec0c88850ff85b4 P 4cc73560bb1d44959f11d6b229819f83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc73560bb1d44959f11d6b229819f83" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 44453 } }
I20240502 20:45:39.184720 24536 leader_election.cc:302] T ad2c408894064c3cbec0c88850ff85b4 P 4cc73560bb1d44959f11d6b229819f83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cc73560bb1d44959f11d6b229819f83; no voters:
I20240502 20:45:39.184873 24536 leader_election.cc:288] T ad2c408894064c3cbec0c88850ff85b4 P 4cc73560bb1d44959f11d6b229819f83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:39.185144 24536 ts_tablet_manager.cc:1398] T ad2c408894064c3cbec0c88850ff85b4 P 4cc73560bb1d44959f11d6b229819f83: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:45:39.186115 24036 catalog_manager.cc:5484] T 9bcbaf8c66744e948862d0adcce47d2a P 64542b4dcd724f9cad68e8ba06e2b1c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64542b4dcd724f9cad68e8ba06e2b1c1 (127.22.104.129). New cstate: current_term: 1 leader_uuid: "64542b4dcd724f9cad68e8ba06e2b1c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64542b4dcd724f9cad68e8ba06e2b1c1" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41685 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:39.186115 24035 catalog_manager.cc:5484] T b3ffa316dc07401e8ffc1a1a521a5ed3 P aa5824746b5d46319a368bf4e861c19e reported cstate change: term changed from 0 to 1, leader changed from <none> to aa5824746b5d46319a368bf4e861c19e (127.22.104.131). New cstate: current_term: 1 leader_uuid: "aa5824746b5d46319a368bf4e861c19e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa5824746b5d46319a368bf4e861c19e" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 39511 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:39.192221 24548 raft_consensus.cc:2781] T ad2c408894064c3cbec0c88850ff85b4 P 4cc73560bb1d44959f11d6b229819f83 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:39.192442 24548 raft_consensus.cc:686] T ad2c408894064c3cbec0c88850ff85b4 P 4cc73560bb1d44959f11d6b229819f83 [term 1 LEADER]: Becoming Leader. State: Replica: 4cc73560bb1d44959f11d6b229819f83, State: Running, Role: LEADER
I20240502 20:45:39.192580 24548 consensus_queue.cc:237] T ad2c408894064c3cbec0c88850ff85b4 P 4cc73560bb1d44959f11d6b229819f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc73560bb1d44959f11d6b229819f83" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 44453 } }
I20240502 20:45:39.196055 24035 catalog_manager.cc:5484] T ad2c408894064c3cbec0c88850ff85b4 P 4cc73560bb1d44959f11d6b229819f83 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cc73560bb1d44959f11d6b229819f83 (127.22.104.130). New cstate: current_term: 1 leader_uuid: "4cc73560bb1d44959f11d6b229819f83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc73560bb1d44959f11d6b229819f83" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 44453 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:39.200140 22946 master.cc:537] Master@127.22.104.190:45795 shutting down...
I20240502 20:45:39.202728 22946 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:45:39.202888 22946 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:39.202939 22946 tablet_replica.cc:330] T 00000000000000000000000000000000 P a0199d692a1540deb1233383a809f02b: stopping tablet replica
I20240502 20:45:39.216240 22946 master.cc:552] Master@127.22.104.190:45795 shutdown complete.
W20240502 20:45:40.193272 24325 heartbeater.cc:644] Failed to heartbeat to 127.22.104.190:45795 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.104.190:45795: connect: Connection refused (error 111)
I20240502 20:45:40.699759 24557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0199d692a1540deb1233383a809f02b)
I20240502 20:45:40.699787 24558 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0199d692a1540deb1233383a809f02b)
I20240502 20:45:40.699952 24558 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0199d692a1540deb1233383a809f02b" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } } peers { permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } } peers { permanent_uuid: "75223d5a730946b68707abb230b536d0" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } }
I20240502 20:45:40.699936 24557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0199d692a1540deb1233383a809f02b" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } } peers { permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } } peers { permanent_uuid: "75223d5a730946b68707abb230b536d0" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } }
I20240502 20:45:40.700394 24557 leader_election.cc:288] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0199d692a1540deb1233383a809f02b (127.22.104.190:45795), 0ed633f4a06e4f85bdc8a892ba118320 (127.22.104.189:44493)
I20240502 20:45:40.700419 24558 leader_election.cc:288] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0199d692a1540deb1233383a809f02b (127.22.104.190:45795), 75223d5a730946b68707abb230b536d0 (127.22.104.188:37977)
I20240502 20:45:40.700835 24182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ed633f4a06e4f85bdc8a892ba118320" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "75223d5a730946b68707abb230b536d0" is_pre_election: true
I20240502 20:45:40.700955 24182 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed633f4a06e4f85bdc8a892ba118320 in term 1.
I20240502 20:45:40.701026 24116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "75223d5a730946b68707abb230b536d0" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "0ed633f4a06e4f85bdc8a892ba118320" is_pre_election: true
I20240502 20:45:40.701138 24116 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75223d5a730946b68707abb230b536d0 in term 1.
I20240502 20:45:40.701366 24091 leader_election.cc:302] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed633f4a06e4f85bdc8a892ba118320, 75223d5a730946b68707abb230b536d0; no voters:
I20240502 20:45:40.701534 24158 leader_election.cc:302] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed633f4a06e4f85bdc8a892ba118320, 75223d5a730946b68707abb230b536d0; no voters:
I20240502 20:45:40.701617 24558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 20:45:40.701660 24558 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0199d692a1540deb1233383a809f02b)
I20240502 20:45:40.701702 24558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:40.701746 24557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240502 20:45:40.701772 24158 leader_election.cc:334] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a0199d692a1540deb1233383a809f02b (127.22.104.190:45795): Network error: Client connection negotiation failed: client connection to 127.22.104.190:45795: connect: Connection refused (error 111)
I20240502 20:45:40.701783 24557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0199d692a1540deb1233383a809f02b)
W20240502 20:45:40.701841 24092 leader_election.cc:334] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a0199d692a1540deb1233383a809f02b (127.22.104.190:45795): Network error: Client connection negotiation failed: client connection to 127.22.104.190:45795: connect: Connection refused (error 111)
I20240502 20:45:40.701869 24557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:40.702822 24558 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0199d692a1540deb1233383a809f02b" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } } peers { permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } } peers { permanent_uuid: "75223d5a730946b68707abb230b536d0" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } }
I20240502 20:45:40.702822 24557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0199d692a1540deb1233383a809f02b" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } } peers { permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } } peers { permanent_uuid: "75223d5a730946b68707abb230b536d0" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } }
I20240502 20:45:40.703200 24557 leader_election.cc:288] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [CANDIDATE]: Term 2 election: Requested vote from peers a0199d692a1540deb1233383a809f02b (127.22.104.190:45795), 0ed633f4a06e4f85bdc8a892ba118320 (127.22.104.189:44493)
I20240502 20:45:40.703438 24558 leader_election.cc:288] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [CANDIDATE]: Term 2 election: Requested vote from peers a0199d692a1540deb1233383a809f02b (127.22.104.190:45795), 75223d5a730946b68707abb230b536d0 (127.22.104.188:37977)
I20240502 20:45:40.703537 24182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ed633f4a06e4f85bdc8a892ba118320" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "75223d5a730946b68707abb230b536d0"
I20240502 20:45:40.703646 24116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "75223d5a730946b68707abb230b536d0" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "0ed633f4a06e4f85bdc8a892ba118320"
I20240502 20:45:40.703759 24116 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 75223d5a730946b68707abb230b536d0 in current term 2: Already voted for candidate 0ed633f4a06e4f85bdc8a892ba118320 in this term.
I20240502 20:45:40.703655 24182 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ed633f4a06e4f85bdc8a892ba118320 in current term 2: Already voted for candidate 75223d5a730946b68707abb230b536d0 in this term.
W20240502 20:45:40.704106 24092 leader_election.cc:334] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a0199d692a1540deb1233383a809f02b (127.22.104.190:45795): Network error: Client connection negotiation failed: client connection to 127.22.104.190:45795: connect: Connection refused (error 111)
I20240502 20:45:40.704324 24091 leader_election.cc:302] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed633f4a06e4f85bdc8a892ba118320; no voters: 75223d5a730946b68707abb230b536d0, a0199d692a1540deb1233383a809f02b
W20240502 20:45:40.704399 24158 leader_election.cc:334] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a0199d692a1540deb1233383a809f02b (127.22.104.190:45795): Network error: Client connection negotiation failed: client connection to 127.22.104.190:45795: connect: Connection refused (error 111)
I20240502 20:45:40.704460 24158 leader_election.cc:302] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 75223d5a730946b68707abb230b536d0; no voters: 0ed633f4a06e4f85bdc8a892ba118320, a0199d692a1540deb1233383a809f02b
I20240502 20:45:40.704515 24558 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 20:45:40.704596 24557 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 20:45:42.317387 24570 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:42.317543 24570 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0199d692a1540deb1233383a809f02b" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } } peers { permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } } peers { permanent_uuid: "75223d5a730946b68707abb230b536d0" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } }
I20240502 20:45:42.317991 24570 leader_election.cc:288] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0199d692a1540deb1233383a809f02b (127.22.104.190:45795), 0ed633f4a06e4f85bdc8a892ba118320 (127.22.104.189:44493)
W20240502 20:45:42.318583 24158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.104.190:45795: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 20:45:42.318856 24116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "75223d5a730946b68707abb230b536d0" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "0ed633f4a06e4f85bdc8a892ba118320" is_pre_election: true
I20240502 20:45:42.318969 24116 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75223d5a730946b68707abb230b536d0 in term 2.
I20240502 20:45:42.319295 24158 leader_election.cc:302] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed633f4a06e4f85bdc8a892ba118320, 75223d5a730946b68707abb230b536d0; no voters:
W20240502 20:45:42.319447 24158 leader_election.cc:334] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0199d692a1540deb1233383a809f02b (127.22.104.190:45795): Network error: Client connection negotiation failed: client connection to 127.22.104.190:45795: connect: Connection refused (error 111)
I20240502 20:45:42.319465 24570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 20:45:42.319517 24570 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:42.319561 24570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:45:42.320513 24570 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0199d692a1540deb1233383a809f02b" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } } peers { permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } } peers { permanent_uuid: "75223d5a730946b68707abb230b536d0" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } }
I20240502 20:45:42.320850 24570 leader_election.cc:288] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [CANDIDATE]: Term 3 election: Requested vote from peers a0199d692a1540deb1233383a809f02b (127.22.104.190:45795), 0ed633f4a06e4f85bdc8a892ba118320 (127.22.104.189:44493)
I20240502 20:45:42.321353 24116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "75223d5a730946b68707abb230b536d0" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "0ed633f4a06e4f85bdc8a892ba118320"
I20240502 20:45:42.321472 24116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [term 2 FOLLOWER]: Advancing to term 3
W20240502 20:45:42.321739 24158 leader_election.cc:334] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a0199d692a1540deb1233383a809f02b (127.22.104.190:45795): Network error: Client connection negotiation failed: client connection to 127.22.104.190:45795: connect: Connection refused (error 111)
I20240502 20:45:42.322336 24116 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75223d5a730946b68707abb230b536d0 in term 3.
I20240502 20:45:42.322589 24158 leader_election.cc:302] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed633f4a06e4f85bdc8a892ba118320, 75223d5a730946b68707abb230b536d0; no voters: a0199d692a1540deb1233383a809f02b
I20240502 20:45:42.322757 24570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 20:45:42.322948 24570 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [term 3 LEADER]: Becoming Leader. State: Replica: 75223d5a730946b68707abb230b536d0, State: Running, Role: LEADER
I20240502 20:45:42.323093 24570 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0199d692a1540deb1233383a809f02b" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } } peers { permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } } peers { permanent_uuid: "75223d5a730946b68707abb230b536d0" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } }
I20240502 20:45:42.323686 24573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75223d5a730946b68707abb230b536d0. Latest consensus state: current_term: 3 leader_uuid: "75223d5a730946b68707abb230b536d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0199d692a1540deb1233383a809f02b" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } } peers { permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } } peers { permanent_uuid: "75223d5a730946b68707abb230b536d0" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } } }
I20240502 20:45:42.323809 24573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:42.324084 24575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 20:45:42.325191 24575 catalog_manager.cc:669] Loaded metadata for table test-workload [id=f273f31e38fb44d5b24a754fae79d491]
I20240502 20:45:42.325932 24575 tablet_loader.cc:96] loaded metadata for tablet 9bcbaf8c66744e948862d0adcce47d2a (table test-workload [id=f273f31e38fb44d5b24a754fae79d491])
I20240502 20:45:42.326133 24575 tablet_loader.cc:96] loaded metadata for tablet ad2c408894064c3cbec0c88850ff85b4 (table test-workload [id=f273f31e38fb44d5b24a754fae79d491])
I20240502 20:45:42.326297 24575 tablet_loader.cc:96] loaded metadata for tablet b3ffa316dc07401e8ffc1a1a521a5ed3 (table test-workload [id=f273f31e38fb44d5b24a754fae79d491])
I20240502 20:45:42.326391 24575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 20:45:42.326810 24575 catalog_manager.cc:1259] Loaded cluster ID: 7dce8b09fcdd4dd496c043b6709256ff
I20240502 20:45:42.326854 24575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 20:45:42.327409 24575 catalog_manager.cc:1504] Loading token signing keys...
I20240502 20:45:42.327800 24575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0: Loaded TSK: 0
I20240502 20:45:42.328289 24575 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 20:45:42.737643 24116 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [term 3 FOLLOWER]: Refusing update from remote peer 75223d5a730946b68707abb230b536d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 20:45:42.738113 24573 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:45:42.739292 24576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75223d5a730946b68707abb230b536d0. Latest consensus state: current_term: 3 leader_uuid: "75223d5a730946b68707abb230b536d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0199d692a1540deb1233383a809f02b" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } } peers { permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } } peers { permanent_uuid: "75223d5a730946b68707abb230b536d0" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } } }
I20240502 20:45:42.739404 24576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 20:45:42.740253 24570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "75223d5a730946b68707abb230b536d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0199d692a1540deb1233383a809f02b" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } } peers { permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } } peers { permanent_uuid: "75223d5a730946b68707abb230b536d0" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } } }
I20240502 20:45:42.740379 24570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:42.740701 24576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "75223d5a730946b68707abb230b536d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0199d692a1540deb1233383a809f02b" member_type: VOTER last_known_addr { host: "127.22.104.190" port: 45795 } } peers { permanent_uuid: "0ed633f4a06e4f85bdc8a892ba118320" member_type: VOTER last_known_addr { host: "127.22.104.189" port: 44493 } } peers { permanent_uuid: "75223d5a730946b68707abb230b536d0" member_type: VOTER last_known_addr { host: "127.22.104.188" port: 37977 } } }
I20240502 20:45:42.740808 24576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [sys.catalog]: This master's current role is: FOLLOWER
W20240502 20:45:42.741228 24158 consensus_peers.cc:485] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 -> Peer a0199d692a1540deb1233383a809f02b (127.22.104.190:45795): Couldn't send request to peer a0199d692a1540deb1233383a809f02b. Status: Network error: Client connection negotiation failed: client connection to 127.22.104.190:45795: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:42.822922 24249 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:45:42.823040 24249 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:45:42.843691 22946 tablet_server.cc:178] TabletServer@127.22.104.129:0 shutting down...
I20240502 20:45:42.847165 22946 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 20:45:42.847357 22946 tablet_replica.cc:330] T 9bcbaf8c66744e948862d0adcce47d2a P 64542b4dcd724f9cad68e8ba06e2b1c1: stopping tablet replica
I20240502 20:45:42.847466 22946 raft_consensus.cc:2229] T 9bcbaf8c66744e948862d0adcce47d2a P 64542b4dcd724f9cad68e8ba06e2b1c1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:45:42.847587 22946 raft_consensus.cc:2258] T 9bcbaf8c66744e948862d0adcce47d2a P 64542b4dcd724f9cad68e8ba06e2b1c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:42.860039 22946 tablet_server.cc:194] TabletServer@127.22.104.129:0 shutdown complete.
I20240502 20:45:42.861999 22946 tablet_server.cc:178] TabletServer@127.22.104.130:0 shutting down...
I20240502 20:45:42.866125 22946 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 20:45:42.866292 22946 tablet_replica.cc:330] T ad2c408894064c3cbec0c88850ff85b4 P 4cc73560bb1d44959f11d6b229819f83: stopping tablet replica
I20240502 20:45:42.866382 22946 raft_consensus.cc:2229] T ad2c408894064c3cbec0c88850ff85b4 P 4cc73560bb1d44959f11d6b229819f83 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:45:42.866489 22946 raft_consensus.cc:2258] T ad2c408894064c3cbec0c88850ff85b4 P 4cc73560bb1d44959f11d6b229819f83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:42.878723 22946 tablet_server.cc:194] TabletServer@127.22.104.130:0 shutdown complete.
I20240502 20:45:42.880578 22946 tablet_server.cc:178] TabletServer@127.22.104.131:0 shutting down...
I20240502 20:45:42.883864 22946 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 20:45:42.884063 22946 tablet_replica.cc:330] T b3ffa316dc07401e8ffc1a1a521a5ed3 P aa5824746b5d46319a368bf4e861c19e: stopping tablet replica
I20240502 20:45:42.884195 22946 raft_consensus.cc:2229] T b3ffa316dc07401e8ffc1a1a521a5ed3 P aa5824746b5d46319a368bf4e861c19e [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:45:42.884313 22946 raft_consensus.cc:2258] T b3ffa316dc07401e8ffc1a1a521a5ed3 P aa5824746b5d46319a368bf4e861c19e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:42.896433 22946 tablet_server.cc:194] TabletServer@127.22.104.131:0 shutdown complete.
I20240502 20:45:42.898278 22946 master.cc:537] Master@127.22.104.189:44493 shutting down...
I20240502 20:45:42.901364 22946 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:42.901494 22946 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:42.901547 22946 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0ed633f4a06e4f85bdc8a892ba118320: stopping tablet replica
I20240502 20:45:42.914121 22946 master.cc:552] Master@127.22.104.189:44493 shutdown complete.
I20240502 20:45:42.916622 22946 master.cc:537] Master@127.22.104.188:37977 shutting down...
I20240502 20:45:42.919907 22946 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [term 3 LEADER]: Raft consensus shutting down.
I20240502 20:45:42.920133 22946 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:42.920269 22946 tablet_replica.cc:330] T 00000000000000000000000000000000 P 75223d5a730946b68707abb230b536d0: stopping tablet replica
I20240502 20:45:42.932714 22946 master.cc:552] Master@127.22.104.188:37977 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (6219 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20240502 20:45:42.939534 22946 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.22.104.190:44783
I20240502 20:45:42.939832 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:42.941896 24584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:42.942366 24581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:42.942868 24582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:42.943289 22946 server_base.cc:1034] running on GCE node
I20240502 20:45:42.943413 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:45:42.943478 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:45:42.943511 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682742943510 us; error 0 us; skew 500 ppm
I20240502 20:45:42.947876 22946 webserver.cc:460] Webserver started at http://127.22.104.190:39389/ using document root <none> and password file <none>
I20240502 20:45:42.948020 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:42.948067 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:42.948141 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:42.948500 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714682726530369-22946-0/minicluster-data/master-0-root/instance:
uuid: "484fa07c9f8a44b98063a3d50078d63b"
format_stamp: "Formatted at 2024-05-02 20:45:42 on dist-test-slave-rgns"
I20240502 20:45:42.949818 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:42.950652 24589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:42.950945 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:42.951011 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714682726530369-22946-0/minicluster-data/master-0-root
uuid: "484fa07c9f8a44b98063a3d50078d63b"
format_stamp: "Formatted at 2024-05-02 20:45:42 on dist-test-slave-rgns"
I20240502 20:45:42.951066 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714682726530369-22946-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714682726530369-22946-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714682726530369-22946-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:42.962764 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:42.963055 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:42.966604 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.190:44783
I20240502 20:45:42.968966 24641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.190:44783 every 8 connection(s)
I20240502 20:45:42.969450 24642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:42.971261 24642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 484fa07c9f8a44b98063a3d50078d63b: Bootstrap starting.
I20240502 20:45:42.971959 24642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 484fa07c9f8a44b98063a3d50078d63b: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:42.972867 24642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 484fa07c9f8a44b98063a3d50078d63b: No bootstrap required, opened a new log
I20240502 20:45:42.973214 24642 raft_consensus.cc:348] T 00000000000000000000000000000000 P 484fa07c9f8a44b98063a3d50078d63b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484fa07c9f8a44b98063a3d50078d63b" member_type: VOTER }
I20240502 20:45:42.973296 24642 raft_consensus.cc:374] T 00000000000000000000000000000000 P 484fa07c9f8a44b98063a3d50078d63b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:42.973325 24642 raft_consensus.cc:729] T 00000000000000000000000000000000 P 484fa07c9f8a44b98063a3d50078d63b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 484fa07c9f8a44b98063a3d50078d63b, State: Initialized, Role: FOLLOWER
I20240502 20:45:42.973438 24642 consensus_queue.cc:260] T 00000000000000000000000000000000 P 484fa07c9f8a44b98063a3d50078d63b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484fa07c9f8a44b98063a3d50078d63b" member_type: VOTER }
I20240502 20:45:42.973493 24642 raft_consensus.cc:388] T 00000000000000000000000000000000 P 484fa07c9f8a44b98063a3d50078d63b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:42.973518 24642 raft_consensus.cc:482] T 00000000000000000000000000000000 P 484fa07c9f8a44b98063a3d50078d63b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:42.973551 24642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 484fa07c9f8a44b98063a3d50078d63b [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:42.974135 24642 raft_consensus.cc:504] T 00000000000000000000000000000000 P 484fa07c9f8a44b98063a3d50078d63b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484fa07c9f8a44b98063a3d50078d63b" member_type: VOTER }
I20240502 20:45:42.974242 24642 leader_election.cc:302] T 00000000000000000000000000000000 P 484fa07c9f8a44b98063a3d50078d63b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 484fa07c9f8a44b98063a3d50078d63b; no voters:
I20240502 20:45:42.974380 24642 leader_election.cc:288] T 00000000000000000000000000000000 P 484fa07c9f8a44b98063a3d50078d63b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:42.974488 24645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 484fa07c9f8a44b98063a3d50078d63b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:42.974681 24645 raft_consensus.cc:686] T 00000000000000000000000000000000 P 484fa07c9f8a44b98063a3d50078d63b [term 1 LEADER]: Becoming Leader. State: Replica: 484fa07c9f8a44b98063a3d50078d63b, State: Running, Role: LEADER
I20240502 20:45:42.974818 24642 sys_catalog.cc:564] T 00000000000000000000000000000000 P 484fa07c9f8a44b98063a3d50078d63b [sys.catalog]: configured and running, proceeding with master startup.
I20240502 20:45:42.974859 24645 consensus_queue.cc:237] T 00000000000000000000000000000000 P 484fa07c9f8a44b98063a3d50078d63b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484fa07c9f8a44b98063a3d50078d63b" member_type: VOTER }
I20240502 20:45:42.975266 24646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 484fa07c9f8a44b98063a3d50078d63b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "484fa07c9f8a44b98063a3d50078d63b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484fa07c9f8a44b98063a3d50078d63b" member_type: VOTER } }
I20240502 20:45:42.975307 24647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 484fa07c9f8a44b98063a3d50078d63b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 484fa07c9f8a44b98063a3d50078d63b. Latest consensus state: current_term: 1 leader_uuid: "484fa07c9f8a44b98063a3d50078d63b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484fa07c9f8a44b98063a3d50078d63b" member_type: VOTER } }
I20240502 20:45:42.975401 24647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 484fa07c9f8a44b98063a3d50078d63b [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:42.975358 24646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 484fa07c9f8a44b98063a3d50078d63b [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:42.975929 24652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 20:45:42.976812 24652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 20:45:42.977043 22946 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 20:45:42.978550 24652 catalog_manager.cc:1347] Generated new cluster ID: 348704f8dacb45bdae804ffbc3a92993
I20240502 20:45:42.978597 24652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 20:45:43.004228 24652 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 20:45:43.004783 24652 catalog_manager.cc:1504] Loading token signing keys...
I20240502 20:45:43.009971 24652 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 484fa07c9f8a44b98063a3d50078d63b: Generated new TSK 0
I20240502 20:45:43.010128 24652 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 20:45:43.041669 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:43.044168 24664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:43.044713 24665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:43.045120 24667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:43.045490 22946 server_base.cc:1034] running on GCE node
I20240502 20:45:43.045686 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:45:43.045734 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:45:43.045758 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682743045758 us; error 0 us; skew 500 ppm
I20240502 20:45:43.046896 22946 webserver.cc:460] Webserver started at http://127.22.104.129:37723/ using document root <none> and password file <none>
I20240502 20:45:43.047072 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:43.047132 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:43.047211 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:43.047626 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714682726530369-22946-0/minicluster-data/ts-0-root/instance:
uuid: "37075c6d145b41bcbab37e6198e4be53"
format_stamp: "Formatted at 2024-05-02 20:45:43 on dist-test-slave-rgns"
I20240502 20:45:43.049060 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 20:45:43.050117 24672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:43.050395 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:43.050465 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714682726530369-22946-0/minicluster-data/ts-0-root
uuid: "37075c6d145b41bcbab37e6198e4be53"
format_stamp: "Formatted at 2024-05-02 20:45:43 on dist-test-slave-rgns"
I20240502 20:45:43.050531 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714682726530369-22946-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714682726530369-22946-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714682726530369-22946-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:43.077041 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:43.077446 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:43.077744 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:43.078224 22946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:43.078264 22946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:43.078307 22946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:43.078334 22946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:43.082233 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.129:35829
I20240502 20:45:43.082264 24735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.129:35829 every 8 connection(s)
I20240502 20:45:43.083006 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:43.085037 24740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:43.085455 24741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:43.085832 24743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:43.086549 22946 server_base.cc:1034] running on GCE node
I20240502 20:45:43.086807 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:45:43.086860 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:45:43.086881 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682743086881 us; error 0 us; skew 500 ppm
I20240502 20:45:43.087563 24736 heartbeater.cc:347] Connected to a master server at 127.22.104.190:44783
I20240502 20:45:43.087671 24736 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:43.087908 24736 heartbeater.cc:511] Master 127.22.104.190:44783 requested a full tablet report, sending...
I20240502 20:45:43.087917 22946 webserver.cc:460] Webserver started at http://127.22.104.130:40019/ using document root <none> and password file <none>
I20240502 20:45:43.088096 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:43.088155 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:43.088222 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:43.088585 24606 ts_manager.cc:194] Registered new tserver with Master: 37075c6d145b41bcbab37e6198e4be53 (127.22.104.129:35829)
I20240502 20:45:43.088702 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714682726530369-22946-0/minicluster-data/ts-1-root/instance:
uuid: "269b176be08c4e5c920987c310eb1bdf"
format_stamp: "Formatted at 2024-05-02 20:45:43 on dist-test-slave-rgns"
I20240502 20:45:43.089576 24606 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49066
I20240502 20:45:43.090382 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 20:45:43.091224 24749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:43.091415 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240502 20:45:43.091480 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714682726530369-22946-0/minicluster-data/ts-1-root
uuid: "269b176be08c4e5c920987c310eb1bdf"
format_stamp: "Formatted at 2024-05-02 20:45:43 on dist-test-slave-rgns"
I20240502 20:45:43.091535 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714682726530369-22946-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714682726530369-22946-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714682726530369-22946-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:43.142546 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:43.142971 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:43.143253 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:43.143756 22946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:43.143798 22946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:43.143833 22946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:43.143862 22946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:43.147715 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.130:45543
I20240502 20:45:43.147789 24812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.130:45543 every 8 connection(s)
I20240502 20:45:43.148542 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:43.150662 24816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:43.151170 24817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:43.151541 24819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:43.152235 22946 server_base.cc:1034] running on GCE node
I20240502 20:45:43.152432 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:45:43.152472 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:45:43.152487 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682743152486 us; error 0 us; skew 500 ppm
I20240502 20:45:43.152701 24813 heartbeater.cc:347] Connected to a master server at 127.22.104.190:44783
I20240502 20:45:43.152798 24813 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:43.153005 24813 heartbeater.cc:511] Master 127.22.104.190:44783 requested a full tablet report, sending...
I20240502 20:45:43.153461 22946 webserver.cc:460] Webserver started at http://127.22.104.131:41603/ using document root <none> and password file <none>
I20240502 20:45:43.153609 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:43.153620 24606 ts_manager.cc:194] Registered new tserver with Master: 269b176be08c4e5c920987c310eb1bdf (127.22.104.130:45543)
I20240502 20:45:43.153651 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:43.153754 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:43.154134 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714682726530369-22946-0/minicluster-data/ts-2-root/instance:
uuid: "396d91efebd6482d84ac0c5acaccd1f0"
format_stamp: "Formatted at 2024-05-02 20:45:43 on dist-test-slave-rgns"
I20240502 20:45:43.154167 24606 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49068
I20240502 20:45:43.155555 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 20:45:43.156463 24824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:43.156679 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:43.156742 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714682726530369-22946-0/minicluster-data/ts-2-root
uuid: "396d91efebd6482d84ac0c5acaccd1f0"
format_stamp: "Formatted at 2024-05-02 20:45:43 on dist-test-slave-rgns"
I20240502 20:45:43.156809 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714682726530369-22946-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714682726530369-22946-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714682726530369-22946-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:43.169693 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:43.170094 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:43.170384 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:43.170868 22946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:43.170910 22946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:43.170953 22946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:43.170980 22946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:43.175215 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.131:45041
I20240502 20:45:43.175324 24891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.131:45041 every 8 connection(s)
I20240502 20:45:43.179684 24892 heartbeater.cc:347] Connected to a master server at 127.22.104.190:44783
I20240502 20:45:43.179801 24892 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:43.180061 24892 heartbeater.cc:511] Master 127.22.104.190:44783 requested a full tablet report, sending...
I20240502 20:45:43.180802 24606 ts_manager.cc:194] Registered new tserver with Master: 396d91efebd6482d84ac0c5acaccd1f0 (127.22.104.131:45041)
I20240502 20:45:43.181216 22946 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.0056015s
I20240502 20:45:43.181391 24606 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49072
I20240502 20:45:44.090538 24736 heartbeater.cc:503] Master 127.22.104.190:44783 was elected leader, sending a full tablet report...
I20240502 20:45:44.155028 24813 heartbeater.cc:503] Master 127.22.104.190:44783 was elected leader, sending a full tablet report...
I20240502 20:45:44.182256 24892 heartbeater.cc:503] Master 127.22.104.190:44783 was elected leader, sending a full tablet report...
I20240502 20:45:44.207319 22946 test_util.cc:268] Using random seed: -1344342063
I20240502 20:45:44.214812 24606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49088:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 20:45:44.215294 24606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 20:45:44.223788 24852 tablet_service.cc:1467] Processing CreateTablet for tablet 53facf0cd4384bae83d638da0d7f4e86 (DEFAULT_TABLE table=test-workload [id=2d0bafac3ce443af8ee17139a2a3afec]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 20:45:44.224102 24852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53facf0cd4384bae83d638da0d7f4e86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:44.225282 24777 tablet_service.cc:1467] Processing CreateTablet for tablet 53facf0cd4384bae83d638da0d7f4e86 (DEFAULT_TABLE table=test-workload [id=2d0bafac3ce443af8ee17139a2a3afec]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 20:45:44.225291 24776 tablet_service.cc:1467] Processing CreateTablet for tablet e811c10fd15d4379bd2c08e8a4e9f1c2 (DEFAULT_TABLE table=test-workload [id=2d0bafac3ce443af8ee17139a2a3afec]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 20:45:44.225606 24777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53facf0cd4384bae83d638da0d7f4e86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:44.225889 24776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e811c10fd15d4379bd2c08e8a4e9f1c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:44.226413 24912 tablet_bootstrap.cc:492] T 53facf0cd4384bae83d638da0d7f4e86 P 396d91efebd6482d84ac0c5acaccd1f0: Bootstrap starting.
I20240502 20:45:44.226539 24851 tablet_service.cc:1467] Processing CreateTablet for tablet e811c10fd15d4379bd2c08e8a4e9f1c2 (DEFAULT_TABLE table=test-workload [id=2d0bafac3ce443af8ee17139a2a3afec]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 20:45:44.226788 24851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e811c10fd15d4379bd2c08e8a4e9f1c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:44.227344 24912 tablet_bootstrap.cc:654] T 53facf0cd4384bae83d638da0d7f4e86 P 396d91efebd6482d84ac0c5acaccd1f0: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:44.228261 24913 tablet_bootstrap.cc:492] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf: Bootstrap starting.
I20240502 20:45:44.228626 24700 tablet_service.cc:1467] Processing CreateTablet for tablet 53facf0cd4384bae83d638da0d7f4e86 (DEFAULT_TABLE table=test-workload [id=2d0bafac3ce443af8ee17139a2a3afec]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 20:45:44.228834 24700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53facf0cd4384bae83d638da0d7f4e86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:44.229001 24913 tablet_bootstrap.cc:654] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:44.229868 24913 tablet_bootstrap.cc:492] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf: No bootstrap required, opened a new log
I20240502 20:45:44.229950 24913 ts_tablet_manager.cc:1367] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:44.230311 24913 raft_consensus.cc:348] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } }
I20240502 20:45:44.230355 24916 tablet_bootstrap.cc:492] T 53facf0cd4384bae83d638da0d7f4e86 P 37075c6d145b41bcbab37e6198e4be53: Bootstrap starting.
I20240502 20:45:44.230504 24699 tablet_service.cc:1467] Processing CreateTablet for tablet e811c10fd15d4379bd2c08e8a4e9f1c2 (DEFAULT_TABLE table=test-workload [id=2d0bafac3ce443af8ee17139a2a3afec]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 20:45:44.230456 24913 raft_consensus.cc:374] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:44.230790 24913 raft_consensus.cc:729] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b176be08c4e5c920987c310eb1bdf, State: Initialized, Role: FOLLOWER
I20240502 20:45:44.230819 24699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e811c10fd15d4379bd2c08e8a4e9f1c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:44.231173 24912 tablet_bootstrap.cc:492] T 53facf0cd4384bae83d638da0d7f4e86 P 396d91efebd6482d84ac0c5acaccd1f0: No bootstrap required, opened a new log
I20240502 20:45:44.231242 24912 ts_tablet_manager.cc:1367] T 53facf0cd4384bae83d638da0d7f4e86 P 396d91efebd6482d84ac0c5acaccd1f0: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 20:45:44.231380 24913 consensus_queue.cc:260] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } }
I20240502 20:45:44.231660 24916 tablet_bootstrap.cc:654] T 53facf0cd4384bae83d638da0d7f4e86 P 37075c6d145b41bcbab37e6198e4be53: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:44.231637 24912 raft_consensus.cc:348] T 53facf0cd4384bae83d638da0d7f4e86 P 396d91efebd6482d84ac0c5acaccd1f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } }
I20240502 20:45:44.231729 24912 raft_consensus.cc:374] T 53facf0cd4384bae83d638da0d7f4e86 P 396d91efebd6482d84ac0c5acaccd1f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:44.231760 24913 ts_tablet_manager.cc:1398] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:44.231760 24912 raft_consensus.cc:729] T 53facf0cd4384bae83d638da0d7f4e86 P 396d91efebd6482d84ac0c5acaccd1f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 396d91efebd6482d84ac0c5acaccd1f0, State: Initialized, Role: FOLLOWER
I20240502 20:45:44.231950 24912 consensus_queue.cc:260] T 53facf0cd4384bae83d638da0d7f4e86 P 396d91efebd6482d84ac0c5acaccd1f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } }
I20240502 20:45:44.232281 24912 ts_tablet_manager.cc:1398] T 53facf0cd4384bae83d638da0d7f4e86 P 396d91efebd6482d84ac0c5acaccd1f0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:44.232309 24913 tablet_bootstrap.cc:492] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 269b176be08c4e5c920987c310eb1bdf: Bootstrap starting.
I20240502 20:45:44.232921 24912 tablet_bootstrap.cc:492] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 396d91efebd6482d84ac0c5acaccd1f0: Bootstrap starting.
I20240502 20:45:44.233151 24913 tablet_bootstrap.cc:654] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 269b176be08c4e5c920987c310eb1bdf: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:44.233927 24912 tablet_bootstrap.cc:654] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 396d91efebd6482d84ac0c5acaccd1f0: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:44.234249 24913 tablet_bootstrap.cc:492] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 269b176be08c4e5c920987c310eb1bdf: No bootstrap required, opened a new log
I20240502 20:45:44.234329 24913 ts_tablet_manager.cc:1367] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 269b176be08c4e5c920987c310eb1bdf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:44.234779 24913 raft_consensus.cc:348] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 269b176be08c4e5c920987c310eb1bdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } }
I20240502 20:45:44.234874 24913 raft_consensus.cc:374] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 269b176be08c4e5c920987c310eb1bdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:44.234903 24913 raft_consensus.cc:729] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 269b176be08c4e5c920987c310eb1bdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b176be08c4e5c920987c310eb1bdf, State: Initialized, Role: FOLLOWER
I20240502 20:45:44.235006 24913 consensus_queue.cc:260] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 269b176be08c4e5c920987c310eb1bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } }
I20240502 20:45:44.235213 24912 tablet_bootstrap.cc:492] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 396d91efebd6482d84ac0c5acaccd1f0: No bootstrap required, opened a new log
I20240502 20:45:44.235301 24912 ts_tablet_manager.cc:1367] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 396d91efebd6482d84ac0c5acaccd1f0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:44.235229 24913 ts_tablet_manager.cc:1398] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 269b176be08c4e5c920987c310eb1bdf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:44.235693 24912 raft_consensus.cc:348] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 396d91efebd6482d84ac0c5acaccd1f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } }
I20240502 20:45:44.235785 24912 raft_consensus.cc:374] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 396d91efebd6482d84ac0c5acaccd1f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:44.235805 24912 raft_consensus.cc:729] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 396d91efebd6482d84ac0c5acaccd1f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 396d91efebd6482d84ac0c5acaccd1f0, State: Initialized, Role: FOLLOWER
I20240502 20:45:44.235901 24912 consensus_queue.cc:260] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 396d91efebd6482d84ac0c5acaccd1f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } }
I20240502 20:45:44.236213 24912 ts_tablet_manager.cc:1398] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 396d91efebd6482d84ac0c5acaccd1f0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:44.236783 24916 tablet_bootstrap.cc:492] T 53facf0cd4384bae83d638da0d7f4e86 P 37075c6d145b41bcbab37e6198e4be53: No bootstrap required, opened a new log
I20240502 20:45:44.236853 24916 ts_tablet_manager.cc:1367] T 53facf0cd4384bae83d638da0d7f4e86 P 37075c6d145b41bcbab37e6198e4be53: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 20:45:44.237171 24916 raft_consensus.cc:348] T 53facf0cd4384bae83d638da0d7f4e86 P 37075c6d145b41bcbab37e6198e4be53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } }
I20240502 20:45:44.237277 24916 raft_consensus.cc:374] T 53facf0cd4384bae83d638da0d7f4e86 P 37075c6d145b41bcbab37e6198e4be53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:44.237320 24916 raft_consensus.cc:729] T 53facf0cd4384bae83d638da0d7f4e86 P 37075c6d145b41bcbab37e6198e4be53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37075c6d145b41bcbab37e6198e4be53, State: Initialized, Role: FOLLOWER
I20240502 20:45:44.237426 24916 consensus_queue.cc:260] T 53facf0cd4384bae83d638da0d7f4e86 P 37075c6d145b41bcbab37e6198e4be53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } }
I20240502 20:45:44.237756 24916 ts_tablet_manager.cc:1398] T 53facf0cd4384bae83d638da0d7f4e86 P 37075c6d145b41bcbab37e6198e4be53: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:44.237923 24916 tablet_bootstrap.cc:492] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 37075c6d145b41bcbab37e6198e4be53: Bootstrap starting.
I20240502 20:45:44.238802 24916 tablet_bootstrap.cc:654] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 37075c6d145b41bcbab37e6198e4be53: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:44.239850 24916 tablet_bootstrap.cc:492] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 37075c6d145b41bcbab37e6198e4be53: No bootstrap required, opened a new log
I20240502 20:45:44.239946 24916 ts_tablet_manager.cc:1367] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 37075c6d145b41bcbab37e6198e4be53: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:45:44.240314 24916 raft_consensus.cc:348] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 37075c6d145b41bcbab37e6198e4be53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } }
I20240502 20:45:44.240393 24916 raft_consensus.cc:374] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 37075c6d145b41bcbab37e6198e4be53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:44.240413 24916 raft_consensus.cc:729] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 37075c6d145b41bcbab37e6198e4be53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37075c6d145b41bcbab37e6198e4be53, State: Initialized, Role: FOLLOWER
I20240502 20:45:44.240496 24916 consensus_queue.cc:260] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 37075c6d145b41bcbab37e6198e4be53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } }
I20240502 20:45:44.240670 24916 ts_tablet_manager.cc:1398] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 37075c6d145b41bcbab37e6198e4be53: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:45:44.242769 24923 raft_consensus.cc:482] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 37075c6d145b41bcbab37e6198e4be53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:44.242846 24923 raft_consensus.cc:504] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 37075c6d145b41bcbab37e6198e4be53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } }
I20240502 20:45:44.243229 24923 leader_election.cc:288] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 37075c6d145b41bcbab37e6198e4be53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b176be08c4e5c920987c310eb1bdf (127.22.104.130:45543), 396d91efebd6482d84ac0c5acaccd1f0 (127.22.104.131:45041)
I20240502 20:45:44.246719 24787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e811c10fd15d4379bd2c08e8a4e9f1c2" candidate_uuid: "37075c6d145b41bcbab37e6198e4be53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b176be08c4e5c920987c310eb1bdf" is_pre_election: true
I20240502 20:45:44.246719 24863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e811c10fd15d4379bd2c08e8a4e9f1c2" candidate_uuid: "37075c6d145b41bcbab37e6198e4be53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "396d91efebd6482d84ac0c5acaccd1f0" is_pre_election: true
I20240502 20:45:44.246866 24863 raft_consensus.cc:2454] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 396d91efebd6482d84ac0c5acaccd1f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37075c6d145b41bcbab37e6198e4be53 in term 0.
I20240502 20:45:44.246866 24787 raft_consensus.cc:2454] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 269b176be08c4e5c920987c310eb1bdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37075c6d145b41bcbab37e6198e4be53 in term 0.
I20240502 20:45:44.247181 24675 leader_election.cc:302] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 37075c6d145b41bcbab37e6198e4be53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 269b176be08c4e5c920987c310eb1bdf, 37075c6d145b41bcbab37e6198e4be53, 396d91efebd6482d84ac0c5acaccd1f0; no voters:
I20240502 20:45:44.247349 24923 raft_consensus.cc:2781] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 37075c6d145b41bcbab37e6198e4be53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:44.247409 24923 raft_consensus.cc:482] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 37075c6d145b41bcbab37e6198e4be53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:44.247437 24923 raft_consensus.cc:3037] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 37075c6d145b41bcbab37e6198e4be53 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:44.248162 24923 raft_consensus.cc:504] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 37075c6d145b41bcbab37e6198e4be53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } }
I20240502 20:45:44.248454 24923 leader_election.cc:288] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 37075c6d145b41bcbab37e6198e4be53 [CANDIDATE]: Term 1 election: Requested vote from peers 269b176be08c4e5c920987c310eb1bdf (127.22.104.130:45543), 396d91efebd6482d84ac0c5acaccd1f0 (127.22.104.131:45041)
I20240502 20:45:44.248749 24787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e811c10fd15d4379bd2c08e8a4e9f1c2" candidate_uuid: "37075c6d145b41bcbab37e6198e4be53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b176be08c4e5c920987c310eb1bdf"
I20240502 20:45:44.248749 24863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e811c10fd15d4379bd2c08e8a4e9f1c2" candidate_uuid: "37075c6d145b41bcbab37e6198e4be53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "396d91efebd6482d84ac0c5acaccd1f0"
I20240502 20:45:44.248878 24787 raft_consensus.cc:3037] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 269b176be08c4e5c920987c310eb1bdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:44.248878 24863 raft_consensus.cc:3037] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 396d91efebd6482d84ac0c5acaccd1f0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:44.249780 24863 raft_consensus.cc:2454] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 396d91efebd6482d84ac0c5acaccd1f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37075c6d145b41bcbab37e6198e4be53 in term 1.
I20240502 20:45:44.249780 24787 raft_consensus.cc:2454] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 269b176be08c4e5c920987c310eb1bdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37075c6d145b41bcbab37e6198e4be53 in term 1.
I20240502 20:45:44.250054 24675 leader_election.cc:302] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 37075c6d145b41bcbab37e6198e4be53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b176be08c4e5c920987c310eb1bdf, 37075c6d145b41bcbab37e6198e4be53; no voters:
I20240502 20:45:44.250190 24923 raft_consensus.cc:2781] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 37075c6d145b41bcbab37e6198e4be53 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:44.250370 24923 raft_consensus.cc:686] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 37075c6d145b41bcbab37e6198e4be53 [term 1 LEADER]: Becoming Leader. State: Replica: 37075c6d145b41bcbab37e6198e4be53, State: Running, Role: LEADER
I20240502 20:45:44.250481 24923 consensus_queue.cc:237] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 37075c6d145b41bcbab37e6198e4be53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } }
I20240502 20:45:44.251955 24606 catalog_manager.cc:5484] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 37075c6d145b41bcbab37e6198e4be53 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37075c6d145b41bcbab37e6198e4be53 (127.22.104.129). New cstate: current_term: 1 leader_uuid: "37075c6d145b41bcbab37e6198e4be53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:44.393661 24917 raft_consensus.cc:482] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:44.393822 24917 raft_consensus.cc:504] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } }
I20240502 20:45:44.394383 24917 leader_election.cc:288] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 396d91efebd6482d84ac0c5acaccd1f0 (127.22.104.131:45041), 37075c6d145b41bcbab37e6198e4be53 (127.22.104.129:35829)
I20240502 20:45:44.397686 24710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53facf0cd4384bae83d638da0d7f4e86" candidate_uuid: "269b176be08c4e5c920987c310eb1bdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37075c6d145b41bcbab37e6198e4be53" is_pre_election: true
I20240502 20:45:44.397663 24863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53facf0cd4384bae83d638da0d7f4e86" candidate_uuid: "269b176be08c4e5c920987c310eb1bdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "396d91efebd6482d84ac0c5acaccd1f0" is_pre_election: true
I20240502 20:45:44.397840 24710 raft_consensus.cc:2454] T 53facf0cd4384bae83d638da0d7f4e86 P 37075c6d145b41bcbab37e6198e4be53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b176be08c4e5c920987c310eb1bdf in term 0.
I20240502 20:45:44.397859 24863 raft_consensus.cc:2454] T 53facf0cd4384bae83d638da0d7f4e86 P 396d91efebd6482d84ac0c5acaccd1f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b176be08c4e5c920987c310eb1bdf in term 0.
I20240502 20:45:44.398106 24750 leader_election.cc:302] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b176be08c4e5c920987c310eb1bdf, 37075c6d145b41bcbab37e6198e4be53; no voters:
I20240502 20:45:44.398262 24917 raft_consensus.cc:2781] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:44.398320 24917 raft_consensus.cc:482] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:44.398363 24917 raft_consensus.cc:3037] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:44.399204 24917 raft_consensus.cc:504] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } }
I20240502 20:45:44.399505 24917 leader_election.cc:288] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [CANDIDATE]: Term 1 election: Requested vote from peers 396d91efebd6482d84ac0c5acaccd1f0 (127.22.104.131:45041), 37075c6d145b41bcbab37e6198e4be53 (127.22.104.129:35829)
I20240502 20:45:44.399794 24863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53facf0cd4384bae83d638da0d7f4e86" candidate_uuid: "269b176be08c4e5c920987c310eb1bdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "396d91efebd6482d84ac0c5acaccd1f0"
I20240502 20:45:44.399811 24710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53facf0cd4384bae83d638da0d7f4e86" candidate_uuid: "269b176be08c4e5c920987c310eb1bdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37075c6d145b41bcbab37e6198e4be53"
I20240502 20:45:44.399878 24863 raft_consensus.cc:3037] T 53facf0cd4384bae83d638da0d7f4e86 P 396d91efebd6482d84ac0c5acaccd1f0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:44.399924 24710 raft_consensus.cc:3037] T 53facf0cd4384bae83d638da0d7f4e86 P 37075c6d145b41bcbab37e6198e4be53 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:44.400559 24863 raft_consensus.cc:2454] T 53facf0cd4384bae83d638da0d7f4e86 P 396d91efebd6482d84ac0c5acaccd1f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b176be08c4e5c920987c310eb1bdf in term 1.
I20240502 20:45:44.400700 24710 raft_consensus.cc:2454] T 53facf0cd4384bae83d638da0d7f4e86 P 37075c6d145b41bcbab37e6198e4be53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b176be08c4e5c920987c310eb1bdf in term 1.
I20240502 20:45:44.400796 24751 leader_election.cc:302] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b176be08c4e5c920987c310eb1bdf, 396d91efebd6482d84ac0c5acaccd1f0; no voters:
I20240502 20:45:44.400956 24917 raft_consensus.cc:2781] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:44.401141 24917 raft_consensus.cc:686] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [term 1 LEADER]: Becoming Leader. State: Replica: 269b176be08c4e5c920987c310eb1bdf, State: Running, Role: LEADER
I20240502 20:45:44.401278 24917 consensus_queue.cc:237] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } }
I20240502 20:45:44.402614 24606 catalog_manager.cc:5484] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf reported cstate change: term changed from 0 to 1, leader changed from <none> to 269b176be08c4e5c920987c310eb1bdf (127.22.104.130). New cstate: current_term: 1 leader_uuid: "269b176be08c4e5c920987c310eb1bdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:44.790735 24933 consensus_queue.cc:1035] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 37075c6d145b41bcbab37e6198e4be53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:44.793918 24933 consensus_queue.cc:1035] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 37075c6d145b41bcbab37e6198e4be53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:44.832270 24917 consensus_queue.cc:1035] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:44.835143 24945 consensus_queue.cc:1035] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:44.976670 24661 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:45:44.977049 24661 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 20:45:44.977124 24661 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:45:46.977423 24661 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:45:46.977860 24661 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 20:45:46.977937 24661 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:45:48.978255 24661 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:45:48.978799 24661 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 20:45:48.978878 24661 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:45:50.979193 24661 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:45:50.979629 24661 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 20:45:50.979704 24661 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:45:52.979964 24661 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:45:52.980414 24661 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 20:45:52.980495 24661 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:45:54.980818 24661 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:45:54.981247 24661 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 20:45:54.981338 24661 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:45:55.458750 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:55.461298 24958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:55.461994 24960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:55.462580 24957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:55.463032 22946 server_base.cc:1034] running on GCE node
I20240502 20:45:55.463212 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:45:55.463248 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:45:55.463263 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682755463262 us; error 0 us; skew 500 ppm
I20240502 20:45:55.464339 22946 webserver.cc:460] Webserver started at http://127.22.104.132:44327/ using document root <none> and password file <none>
I20240502 20:45:55.464507 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:55.464552 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:55.464622 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:55.464987 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714682726530369-22946-0/minicluster-data/ts-3-root/instance:
uuid: "d4b68aa4d13d4453a245cf08f37c7b31"
format_stamp: "Formatted at 2024-05-02 20:45:55 on dist-test-slave-rgns"
I20240502 20:45:55.466450 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:55.467476 24965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:55.467716 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:55.467777 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714682726530369-22946-0/minicluster-data/ts-3-root
uuid: "d4b68aa4d13d4453a245cf08f37c7b31"
format_stamp: "Formatted at 2024-05-02 20:45:55 on dist-test-slave-rgns"
I20240502 20:45:55.467840 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714682726530369-22946-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714682726530369-22946-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714682726530369-22946-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:55.476822 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:55.477130 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:55.477365 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:55.477815 22946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:55.477854 22946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:55.477886 22946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:55.477914 22946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:55.483141 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.132:39841
I20240502 20:45:55.483274 25029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.132:39841 every 8 connection(s)
I20240502 20:45:55.487643 25030 heartbeater.cc:347] Connected to a master server at 127.22.104.190:44783
I20240502 20:45:55.487751 25030 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:55.487968 25030 heartbeater.cc:511] Master 127.22.104.190:44783 requested a full tablet report, sending...
I20240502 20:45:55.488606 24605 ts_manager.cc:194] Registered new tserver with Master: d4b68aa4d13d4453a245cf08f37c7b31 (127.22.104.132:39841)
I20240502 20:45:55.489156 24605 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45190
I20240502 20:45:55.991025 24787 consensus_queue.cc:237] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } attrs { replace: true } } peers { permanent_uuid: "d4b68aa4d13d4453a245cf08f37c7b31" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 39841 } attrs { promote: true } }
I20240502 20:45:55.992532 24710 raft_consensus.cc:1261] T 53facf0cd4384bae83d638da0d7f4e86 P 37075c6d145b41bcbab37e6198e4be53 [term 1 FOLLOWER]: Refusing update from remote peer 269b176be08c4e5c920987c310eb1bdf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:55.992574 24863 raft_consensus.cc:1261] T 53facf0cd4384bae83d638da0d7f4e86 P 396d91efebd6482d84ac0c5acaccd1f0 [term 1 FOLLOWER]: Refusing update from remote peer 269b176be08c4e5c920987c310eb1bdf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:55.992935 24954 consensus_queue.cc:1035] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:45:55.993181 25038 consensus_queue.cc:1035] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:45:55.994854 24954 raft_consensus.cc:2932] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4b68aa4d13d4453a245cf08f37c7b31 (127.22.104.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } attrs { replace: true } } peers { permanent_uuid: "d4b68aa4d13d4453a245cf08f37c7b31" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 39841 } attrs { promote: true } } }
I20240502 20:45:55.995282 24863 raft_consensus.cc:2932] T 53facf0cd4384bae83d638da0d7f4e86 P 396d91efebd6482d84ac0c5acaccd1f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4b68aa4d13d4453a245cf08f37c7b31 (127.22.104.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } attrs { replace: true } } peers { permanent_uuid: "d4b68aa4d13d4453a245cf08f37c7b31" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 39841 } attrs { promote: true } } }
W20240502 20:45:55.996829 24753 consensus_peers.cc:485] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf -> Peer d4b68aa4d13d4453a245cf08f37c7b31 (127.22.104.132:39841): Couldn't send request to peer d4b68aa4d13d4453a245cf08f37c7b31. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53facf0cd4384bae83d638da0d7f4e86. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:55.997454 24605 catalog_manager.cc:5484] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf reported cstate change: config changed from index -1 to 2, NON_VOTER d4b68aa4d13d4453a245cf08f37c7b31 (127.22.104.132) added. New cstate: current_term: 1 leader_uuid: "269b176be08c4e5c920987c310eb1bdf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4b68aa4d13d4453a245cf08f37c7b31" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 39841 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:55.997684 24710 raft_consensus.cc:2932] T 53facf0cd4384bae83d638da0d7f4e86 P 37075c6d145b41bcbab37e6198e4be53 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4b68aa4d13d4453a245cf08f37c7b31 (127.22.104.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } attrs { replace: true } } peers { permanent_uuid: "d4b68aa4d13d4453a245cf08f37c7b31" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 39841 } attrs { promote: true } } }
I20240502 20:45:56.489925 25030 heartbeater.cc:503] Master 127.22.104.190:44783 was elected leader, sending a full tablet report...
I20240502 20:45:56.541077 25048 ts_tablet_manager.cc:902] T 53facf0cd4384bae83d638da0d7f4e86 P d4b68aa4d13d4453a245cf08f37c7b31: Initiating tablet copy from peer 269b176be08c4e5c920987c310eb1bdf (127.22.104.130:45543)
I20240502 20:45:56.541652 25048 tablet_copy_client.cc:308] T 53facf0cd4384bae83d638da0d7f4e86 P d4b68aa4d13d4453a245cf08f37c7b31: tablet copy: Beginning tablet copy session from remote peer at address 127.22.104.130:45543
I20240502 20:45:56.545575 24797 tablet_copy_service.cc:140] P 269b176be08c4e5c920987c310eb1bdf: Received BeginTabletCopySession request for tablet 53facf0cd4384bae83d638da0d7f4e86 from peer d4b68aa4d13d4453a245cf08f37c7b31 ({username='slave'} at 127.0.0.1:58746)
I20240502 20:45:56.545699 24797 tablet_copy_service.cc:161] P 269b176be08c4e5c920987c310eb1bdf: Beginning new tablet copy session on tablet 53facf0cd4384bae83d638da0d7f4e86 from peer d4b68aa4d13d4453a245cf08f37c7b31 at {username='slave'} at 127.0.0.1:58746: session id = d4b68aa4d13d4453a245cf08f37c7b31-53facf0cd4384bae83d638da0d7f4e86
I20240502 20:45:56.546598 24797 tablet_copy_source_session.cc:205] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:56.547449 25048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53facf0cd4384bae83d638da0d7f4e86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:56.549777 25048 tablet_copy_client.cc:783] T 53facf0cd4384bae83d638da0d7f4e86 P d4b68aa4d13d4453a245cf08f37c7b31: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:56.549947 25048 tablet_copy_client.cc:647] T 53facf0cd4384bae83d638da0d7f4e86 P d4b68aa4d13d4453a245cf08f37c7b31: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:56.551232 25048 tablet_copy_client.cc:523] T 53facf0cd4384bae83d638da0d7f4e86 P d4b68aa4d13d4453a245cf08f37c7b31: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:56.552551 25048 tablet_bootstrap.cc:492] T 53facf0cd4384bae83d638da0d7f4e86 P d4b68aa4d13d4453a245cf08f37c7b31: Bootstrap starting.
I20240502 20:45:56.556068 25048 tablet_bootstrap.cc:492] T 53facf0cd4384bae83d638da0d7f4e86 P d4b68aa4d13d4453a245cf08f37c7b31: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:56.556278 25048 tablet_bootstrap.cc:492] T 53facf0cd4384bae83d638da0d7f4e86 P d4b68aa4d13d4453a245cf08f37c7b31: Bootstrap complete.
I20240502 20:45:56.556428 25048 ts_tablet_manager.cc:1367] T 53facf0cd4384bae83d638da0d7f4e86 P d4b68aa4d13d4453a245cf08f37c7b31: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20240502 20:45:56.556864 25048 raft_consensus.cc:348] T 53facf0cd4384bae83d638da0d7f4e86 P d4b68aa4d13d4453a245cf08f37c7b31 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } attrs { replace: true } } peers { permanent_uuid: "d4b68aa4d13d4453a245cf08f37c7b31" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 39841 } attrs { promote: true } }
I20240502 20:45:56.556991 25048 raft_consensus.cc:729] T 53facf0cd4384bae83d638da0d7f4e86 P d4b68aa4d13d4453a245cf08f37c7b31 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4b68aa4d13d4453a245cf08f37c7b31, State: Initialized, Role: LEARNER
I20240502 20:45:56.557103 25048 consensus_queue.cc:260] T 53facf0cd4384bae83d638da0d7f4e86 P d4b68aa4d13d4453a245cf08f37c7b31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } attrs { replace: true } } peers { permanent_uuid: "d4b68aa4d13d4453a245cf08f37c7b31" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 39841 } attrs { promote: true } }
I20240502 20:45:56.557497 25048 ts_tablet_manager.cc:1398] T 53facf0cd4384bae83d638da0d7f4e86 P d4b68aa4d13d4453a245cf08f37c7b31: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 20:45:56.558094 24797 tablet_copy_service.cc:342] P 269b176be08c4e5c920987c310eb1bdf: Request end of tablet copy session d4b68aa4d13d4453a245cf08f37c7b31-53facf0cd4384bae83d638da0d7f4e86 received from {username='slave'} at 127.0.0.1:58746
I20240502 20:45:56.558161 24797 tablet_copy_service.cc:433] P 269b176be08c4e5c920987c310eb1bdf: ending tablet copy session d4b68aa4d13d4453a245cf08f37c7b31-53facf0cd4384bae83d638da0d7f4e86 on tablet 53facf0cd4384bae83d638da0d7f4e86 with peer d4b68aa4d13d4453a245cf08f37c7b31
I20240502 20:45:56.981616 24661 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:45:56.981956 24661 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 20:45:56.982031 24661 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:45:56.990517 25003 raft_consensus.cc:1203] T 53facf0cd4384bae83d638da0d7f4e86 P d4b68aa4d13d4453a245cf08f37c7b31 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 20:45:57.487847 25058 raft_consensus.cc:1050] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf: attempting to promote NON_VOTER d4b68aa4d13d4453a245cf08f37c7b31 to VOTER
I20240502 20:45:57.488401 25058 consensus_queue.cc:237] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } attrs { replace: true } } peers { permanent_uuid: "d4b68aa4d13d4453a245cf08f37c7b31" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 39841 } attrs { promote: false } }
I20240502 20:45:57.489728 24863 raft_consensus.cc:1261] T 53facf0cd4384bae83d638da0d7f4e86 P 396d91efebd6482d84ac0c5acaccd1f0 [term 1 FOLLOWER]: Refusing update from remote peer 269b176be08c4e5c920987c310eb1bdf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:45:57.489732 25003 raft_consensus.cc:1261] T 53facf0cd4384bae83d638da0d7f4e86 P d4b68aa4d13d4453a245cf08f37c7b31 [term 1 LEARNER]: Refusing update from remote peer 269b176be08c4e5c920987c310eb1bdf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:45:57.489761 24710 raft_consensus.cc:1261] T 53facf0cd4384bae83d638da0d7f4e86 P 37075c6d145b41bcbab37e6198e4be53 [term 1 FOLLOWER]: Refusing update from remote peer 269b176be08c4e5c920987c310eb1bdf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:45:57.490175 25057 consensus_queue.cc:1035] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4b68aa4d13d4453a245cf08f37c7b31" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 39841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:45:57.490381 25058 consensus_queue.cc:1035] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:45:57.490603 25060 consensus_queue.cc:1035] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:45:57.492447 25058 raft_consensus.cc:2932] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4b68aa4d13d4453a245cf08f37c7b31 (127.22.104.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } attrs { replace: true } } peers { permanent_uuid: "d4b68aa4d13d4453a245cf08f37c7b31" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 39841 } attrs { promote: false } } }
I20240502 20:45:57.492841 25003 raft_consensus.cc:2932] T 53facf0cd4384bae83d638da0d7f4e86 P d4b68aa4d13d4453a245cf08f37c7b31 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4b68aa4d13d4453a245cf08f37c7b31 (127.22.104.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } attrs { replace: true } } peers { permanent_uuid: "d4b68aa4d13d4453a245cf08f37c7b31" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 39841 } attrs { promote: false } } }
I20240502 20:45:57.494431 24863 raft_consensus.cc:2932] T 53facf0cd4384bae83d638da0d7f4e86 P 396d91efebd6482d84ac0c5acaccd1f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4b68aa4d13d4453a245cf08f37c7b31 (127.22.104.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } attrs { replace: true } } peers { permanent_uuid: "d4b68aa4d13d4453a245cf08f37c7b31" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 39841 } attrs { promote: false } } }
I20240502 20:45:57.494496 24710 raft_consensus.cc:2932] T 53facf0cd4384bae83d638da0d7f4e86 P 37075c6d145b41bcbab37e6198e4be53 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4b68aa4d13d4453a245cf08f37c7b31 (127.22.104.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } attrs { replace: true } } peers { permanent_uuid: "d4b68aa4d13d4453a245cf08f37c7b31" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 39841 } attrs { promote: false } } }
I20240502 20:45:57.494820 24606 catalog_manager.cc:5484] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf reported cstate change: config changed from index 2 to 3, d4b68aa4d13d4453a245cf08f37c7b31 (127.22.104.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "269b176be08c4e5c920987c310eb1bdf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37075c6d145b41bcbab37e6198e4be53" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 35829 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4b68aa4d13d4453a245cf08f37c7b31" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 39841 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:57.498950 24786 consensus_queue.cc:237] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "d4b68aa4d13d4453a245cf08f37c7b31" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 39841 } attrs { promote: false } }
I20240502 20:45:57.499873 25003 raft_consensus.cc:1261] T 53facf0cd4384bae83d638da0d7f4e86 P d4b68aa4d13d4453a245cf08f37c7b31 [term 1 FOLLOWER]: Refusing update from remote peer 269b176be08c4e5c920987c310eb1bdf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:45:57.499889 24863 raft_consensus.cc:1261] T 53facf0cd4384bae83d638da0d7f4e86 P 396d91efebd6482d84ac0c5acaccd1f0 [term 1 FOLLOWER]: Refusing update from remote peer 269b176be08c4e5c920987c310eb1bdf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:45:57.500259 25060 consensus_queue.cc:1035] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4b68aa4d13d4453a245cf08f37c7b31" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 39841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:45:57.500448 25037 consensus_queue.cc:1035] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:45:57.501849 25037 raft_consensus.cc:2932] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37075c6d145b41bcbab37e6198e4be53 (127.22.104.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "d4b68aa4d13d4453a245cf08f37c7b31" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 39841 } attrs { promote: false } } }
I20240502 20:45:57.502117 24863 raft_consensus.cc:2932] T 53facf0cd4384bae83d638da0d7f4e86 P 396d91efebd6482d84ac0c5acaccd1f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37075c6d145b41bcbab37e6198e4be53 (127.22.104.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "d4b68aa4d13d4453a245cf08f37c7b31" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 39841 } attrs { promote: false } } }
I20240502 20:45:57.503332 24592 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 53facf0cd4384bae83d638da0d7f4e86 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:45:57.503841 25003 raft_consensus.cc:2932] T 53facf0cd4384bae83d638da0d7f4e86 P d4b68aa4d13d4453a245cf08f37c7b31 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37075c6d145b41bcbab37e6198e4be53 (127.22.104.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "d4b68aa4d13d4453a245cf08f37c7b31" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 39841 } attrs { promote: false } } }
I20240502 20:45:57.503986 24605 catalog_manager.cc:5484] T 53facf0cd4384bae83d638da0d7f4e86 P 396d91efebd6482d84ac0c5acaccd1f0 reported cstate change: config changed from index 3 to 4, VOTER 37075c6d145b41bcbab37e6198e4be53 (127.22.104.129) evicted. New cstate: current_term: 1 leader_uuid: "269b176be08c4e5c920987c310eb1bdf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "396d91efebd6482d84ac0c5acaccd1f0" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 45041 } } peers { permanent_uuid: "269b176be08c4e5c920987c310eb1bdf" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 45543 } } peers { permanent_uuid: "d4b68aa4d13d4453a245cf08f37c7b31" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 39841 } attrs { promote: false } } }
I20240502 20:45:57.506201 24699 tablet_service.cc:1514] Processing DeleteTablet for tablet 53facf0cd4384bae83d638da0d7f4e86 with delete_type TABLET_DATA_TOMBSTONED (TS 37075c6d145b41bcbab37e6198e4be53 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49728
I20240502 20:45:57.506727 25070 tablet_replica.cc:330] T 53facf0cd4384bae83d638da0d7f4e86 P 37075c6d145b41bcbab37e6198e4be53: stopping tablet replica
I20240502 20:45:57.506858 25070 raft_consensus.cc:2229] T 53facf0cd4384bae83d638da0d7f4e86 P 37075c6d145b41bcbab37e6198e4be53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:57.506968 25070 raft_consensus.cc:2258] T 53facf0cd4384bae83d638da0d7f4e86 P 37075c6d145b41bcbab37e6198e4be53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:57.507467 25070 ts_tablet_manager.cc:1875] T 53facf0cd4384bae83d638da0d7f4e86 P 37075c6d145b41bcbab37e6198e4be53: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:57.509166 25070 ts_tablet_manager.cc:1888] T 53facf0cd4384bae83d638da0d7f4e86 P 37075c6d145b41bcbab37e6198e4be53: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 20:45:57.509238 25070 log.cc:1196] T 53facf0cd4384bae83d638da0d7f4e86 P 37075c6d145b41bcbab37e6198e4be53: Deleting WAL directory at /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714682726530369-22946-0/minicluster-data/ts-0-root/wals/53facf0cd4384bae83d638da0d7f4e86
I20240502 20:45:57.509673 24590 catalog_manager.cc:4830] TS 37075c6d145b41bcbab37e6198e4be53 (127.22.104.129:35829): tablet 53facf0cd4384bae83d638da0d7f4e86 (table test-workload [id=2d0bafac3ce443af8ee17139a2a3afec]) successfully deleted
I20240502 20:45:58.982367 24661 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:45:58.982856 24661 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 20:45:58.982940 24661 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:46:00.983258 24661 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:46:00.983692 24661 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 20:46:00.983775 24661 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:46:02.984138 24661 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:46:02.984587 24661 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 20:46:02.984668 24661 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:46:04.985020 24661 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:46:04.985489 24661 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 20:46:04.985579 24661 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:46:06.985936 24661 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:46:06.986356 24661 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 20:46:06.986435 24661 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:46:08.986802 24661 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:46:08.987216 24661 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 20:46:08.987287 24661 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:46:09.626160 22946 tablet_server.cc:178] TabletServer@127.22.104.129:0 shutting down...
I20240502 20:46:09.630059 22946 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 20:46:09.630292 22946 tablet_replica.cc:330] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 37075c6d145b41bcbab37e6198e4be53: stopping tablet replica
I20240502 20:46:09.630401 22946 raft_consensus.cc:2229] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 37075c6d145b41bcbab37e6198e4be53 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:09.630623 22946 raft_consensus.cc:2258] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 37075c6d145b41bcbab37e6198e4be53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.644667 22946 tablet_server.cc:194] TabletServer@127.22.104.129:0 shutdown complete.
I20240502 20:46:09.647276 22946 tablet_server.cc:178] TabletServer@127.22.104.130:0 shutting down...
I20240502 20:46:09.651746 22946 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 20:46:09.651974 22946 tablet_replica.cc:330] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 269b176be08c4e5c920987c310eb1bdf: stopping tablet replica
I20240502 20:46:09.652094 22946 raft_consensus.cc:2229] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 269b176be08c4e5c920987c310eb1bdf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:09.652210 22946 raft_consensus.cc:2258] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 269b176be08c4e5c920987c310eb1bdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.652647 22946 tablet_replica.cc:330] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf: stopping tablet replica
I20240502 20:46:09.652724 22946 raft_consensus.cc:2229] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:09.652890 22946 raft_consensus.cc:2258] T 53facf0cd4384bae83d638da0d7f4e86 P 269b176be08c4e5c920987c310eb1bdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.665966 22946 tablet_server.cc:194] TabletServer@127.22.104.130:0 shutdown complete.
I20240502 20:46:09.668536 22946 tablet_server.cc:178] TabletServer@127.22.104.131:0 shutting down...
I20240502 20:46:09.672984 22946 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 20:46:09.673249 22946 tablet_replica.cc:330] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 396d91efebd6482d84ac0c5acaccd1f0: stopping tablet replica
I20240502 20:46:09.673396 22946 raft_consensus.cc:2229] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 396d91efebd6482d84ac0c5acaccd1f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:09.673545 22946 raft_consensus.cc:2258] T e811c10fd15d4379bd2c08e8a4e9f1c2 P 396d91efebd6482d84ac0c5acaccd1f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.674049 22946 tablet_replica.cc:330] T 53facf0cd4384bae83d638da0d7f4e86 P 396d91efebd6482d84ac0c5acaccd1f0: stopping tablet replica
I20240502 20:46:09.674144 22946 raft_consensus.cc:2229] T 53facf0cd4384bae83d638da0d7f4e86 P 396d91efebd6482d84ac0c5acaccd1f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:09.674273 22946 raft_consensus.cc:2258] T 53facf0cd4384bae83d638da0d7f4e86 P 396d91efebd6482d84ac0c5acaccd1f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.688211 22946 tablet_server.cc:194] TabletServer@127.22.104.131:0 shutdown complete.
I20240502 20:46:09.690620 22946 tablet_server.cc:178] TabletServer@127.22.104.132:0 shutting down...
I20240502 20:46:09.694836 22946 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 20:46:09.695060 22946 tablet_replica.cc:330] T 53facf0cd4384bae83d638da0d7f4e86 P d4b68aa4d13d4453a245cf08f37c7b31: stopping tablet replica
I20240502 20:46:09.695187 22946 raft_consensus.cc:2229] T 53facf0cd4384bae83d638da0d7f4e86 P d4b68aa4d13d4453a245cf08f37c7b31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:09.695331 22946 raft_consensus.cc:2258] T 53facf0cd4384bae83d638da0d7f4e86 P d4b68aa4d13d4453a245cf08f37c7b31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.709245 22946 tablet_server.cc:194] TabletServer@127.22.104.132:0 shutdown complete.
I20240502 20:46:09.711594 22946 master.cc:537] Master@127.22.104.190:44783 shutting down...
I20240502 20:46:09.715294 22946 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 484fa07c9f8a44b98063a3d50078d63b [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:09.715479 22946 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 484fa07c9f8a44b98063a3d50078d63b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.715553 22946 tablet_replica.cc:330] T 00000000000000000000000000000000 P 484fa07c9f8a44b98063a3d50078d63b: stopping tablet replica
I20240502 20:46:09.728191 22946 master.cc:552] Master@127.22.104.190:44783 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (26795 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20240502 20:46:09.735112 22946 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.22.104.190:41287
I20240502 20:46:09.735455 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:09.737108 25075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:09.737620 25076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:09.738188 25078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:09.738685 22946 server_base.cc:1034] running on GCE node
I20240502 20:46:09.738863 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:46:09.738898 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:46:09.738914 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682769738914 us; error 0 us; skew 500 ppm
I20240502 20:46:09.746078 22946 webserver.cc:460] Webserver started at http://127.22.104.190:36955/ using document root <none> and password file <none>
I20240502 20:46:09.746259 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:09.746315 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:09.746395 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:09.747036 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714682726530369-22946-0/minicluster-data/master-0-root/instance:
uuid: "ddfebc8bb3a445e2ac07f31648f0b439"
format_stamp: "Formatted at 2024-05-02 20:46:09 on dist-test-slave-rgns"
I20240502 20:46:09.748580 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240502 20:46:09.749694 25083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:09.749955 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:09.750020 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714682726530369-22946-0/minicluster-data/master-0-root
uuid: "ddfebc8bb3a445e2ac07f31648f0b439"
format_stamp: "Formatted at 2024-05-02 20:46:09 on dist-test-slave-rgns"
I20240502 20:46:09.750082 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714682726530369-22946-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714682726530369-22946-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714682726530369-22946-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:09.758076 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:09.758435 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:09.762354 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.190:41287
I20240502 20:46:09.764628 25135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.190:41287 every 8 connection(s)
I20240502 20:46:09.765081 25136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.767002 25136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddfebc8bb3a445e2ac07f31648f0b439: Bootstrap starting.
I20240502 20:46:09.767794 25136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddfebc8bb3a445e2ac07f31648f0b439: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:09.768860 25136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddfebc8bb3a445e2ac07f31648f0b439: No bootstrap required, opened a new log
I20240502 20:46:09.769266 25136 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddfebc8bb3a445e2ac07f31648f0b439 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddfebc8bb3a445e2ac07f31648f0b439" member_type: VOTER }
I20240502 20:46:09.769366 25136 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddfebc8bb3a445e2ac07f31648f0b439 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:09.769397 25136 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddfebc8bb3a445e2ac07f31648f0b439 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddfebc8bb3a445e2ac07f31648f0b439, State: Initialized, Role: FOLLOWER
I20240502 20:46:09.769536 25136 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddfebc8bb3a445e2ac07f31648f0b439 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddfebc8bb3a445e2ac07f31648f0b439" member_type: VOTER }
I20240502 20:46:09.769639 25136 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddfebc8bb3a445e2ac07f31648f0b439 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:46:09.769673 25136 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddfebc8bb3a445e2ac07f31648f0b439 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:46:09.769719 25136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddfebc8bb3a445e2ac07f31648f0b439 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:09.770424 25136 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddfebc8bb3a445e2ac07f31648f0b439 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddfebc8bb3a445e2ac07f31648f0b439" member_type: VOTER }
I20240502 20:46:09.770565 25136 leader_election.cc:302] T 00000000000000000000000000000000 P ddfebc8bb3a445e2ac07f31648f0b439 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddfebc8bb3a445e2ac07f31648f0b439; no voters:
I20240502 20:46:09.770804 25136 leader_election.cc:288] T 00000000000000000000000000000000 P ddfebc8bb3a445e2ac07f31648f0b439 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:46:09.770903 25139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddfebc8bb3a445e2ac07f31648f0b439 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:09.771095 25139 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddfebc8bb3a445e2ac07f31648f0b439 [term 1 LEADER]: Becoming Leader. State: Replica: ddfebc8bb3a445e2ac07f31648f0b439, State: Running, Role: LEADER
I20240502 20:46:09.771260 25139 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddfebc8bb3a445e2ac07f31648f0b439 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddfebc8bb3a445e2ac07f31648f0b439" member_type: VOTER }
I20240502 20:46:09.771267 25136 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddfebc8bb3a445e2ac07f31648f0b439 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 20:46:09.771672 25140 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddfebc8bb3a445e2ac07f31648f0b439 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ddfebc8bb3a445e2ac07f31648f0b439" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddfebc8bb3a445e2ac07f31648f0b439" member_type: VOTER } }
I20240502 20:46:09.771782 25140 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddfebc8bb3a445e2ac07f31648f0b439 [sys.catalog]: This master's current role is: LEADER
I20240502 20:46:09.772403 25143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 20:46:09.771687 25141 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddfebc8bb3a445e2ac07f31648f0b439 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddfebc8bb3a445e2ac07f31648f0b439. Latest consensus state: current_term: 1 leader_uuid: "ddfebc8bb3a445e2ac07f31648f0b439" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddfebc8bb3a445e2ac07f31648f0b439" member_type: VOTER } }
I20240502 20:46:09.773150 25141 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddfebc8bb3a445e2ac07f31648f0b439 [sys.catalog]: This master's current role is: LEADER
I20240502 20:46:09.773351 25143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 20:46:09.773545 22946 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 20:46:09.775290 25143 catalog_manager.cc:1347] Generated new cluster ID: ded252d6585c4a7c95ca2ed274bb6d45
I20240502 20:46:09.775341 25143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 20:46:09.792634 25143 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 20:46:09.793244 25143 catalog_manager.cc:1504] Loading token signing keys...
I20240502 20:46:09.799578 25143 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ddfebc8bb3a445e2ac07f31648f0b439: Generated new TSK 0
I20240502 20:46:09.799775 25143 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 20:46:09.805634 22946 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 7.8208e-05s
I20240502 20:46:21.884243 22946 master.cc:537] Master@127.22.104.190:41287 shutting down...
I20240502 20:46:21.887226 22946 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ddfebc8bb3a445e2ac07f31648f0b439 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:21.887413 22946 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ddfebc8bb3a445e2ac07f31648f0b439 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:21.887480 22946 tablet_replica.cc:330] T 00000000000000000000000000000000 P ddfebc8bb3a445e2ac07f31648f0b439: stopping tablet replica
I20240502 20:46:21.899457 22946 master.cc:552] Master@127.22.104.190:41287 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12168 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20240502 20:46:21.903630 22946 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.22.104.190:37589
I20240502 20:46:21.903944 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:21.905877 25158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:21.906549 25161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:21.907158 25159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:21.907647 22946 server_base.cc:1034] running on GCE node
I20240502 20:46:21.907797 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:46:21.907835 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:46:21.907855 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682781907855 us; error 0 us; skew 500 ppm
I20240502 20:46:21.908802 22946 webserver.cc:460] Webserver started at http://127.22.104.190:42219/ using document root <none> and password file <none>
I20240502 20:46:21.908942 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:21.908993 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:21.909068 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:21.909442 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714682726530369-22946-0/minicluster-data/master-0-root/instance:
uuid: "67ad22b014554d79b4b2abf5a553d99c"
format_stamp: "Formatted at 2024-05-02 20:46:21 on dist-test-slave-rgns"
I20240502 20:46:21.910854 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:21.911715 25166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:21.911921 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:21.911984 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714682726530369-22946-0/minicluster-data/master-0-root
uuid: "67ad22b014554d79b4b2abf5a553d99c"
format_stamp: "Formatted at 2024-05-02 20:46:21 on dist-test-slave-rgns"
I20240502 20:46:21.912050 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714682726530369-22946-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714682726530369-22946-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714682726530369-22946-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:21.926616 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:21.927038 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:21.931051 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.190:37589
I20240502 20:46:21.931648 25220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.190:37589 every 8 connection(s)
I20240502 20:46:21.932154 25221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:21.933939 25221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67ad22b014554d79b4b2abf5a553d99c: Bootstrap starting.
I20240502 20:46:21.934893 25221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67ad22b014554d79b4b2abf5a553d99c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:21.935886 25221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67ad22b014554d79b4b2abf5a553d99c: No bootstrap required, opened a new log
I20240502 20:46:21.936240 25221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67ad22b014554d79b4b2abf5a553d99c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ad22b014554d79b4b2abf5a553d99c" member_type: VOTER }
I20240502 20:46:21.936326 25221 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67ad22b014554d79b4b2abf5a553d99c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:21.936358 25221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67ad22b014554d79b4b2abf5a553d99c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67ad22b014554d79b4b2abf5a553d99c, State: Initialized, Role: FOLLOWER
I20240502 20:46:21.936458 25221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67ad22b014554d79b4b2abf5a553d99c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ad22b014554d79b4b2abf5a553d99c" member_type: VOTER }
I20240502 20:46:21.936511 25221 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67ad22b014554d79b4b2abf5a553d99c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:46:21.936538 25221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67ad22b014554d79b4b2abf5a553d99c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:46:21.936578 25221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67ad22b014554d79b4b2abf5a553d99c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:21.937199 25221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67ad22b014554d79b4b2abf5a553d99c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ad22b014554d79b4b2abf5a553d99c" member_type: VOTER }
I20240502 20:46:21.937316 25221 leader_election.cc:302] T 00000000000000000000000000000000 P 67ad22b014554d79b4b2abf5a553d99c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67ad22b014554d79b4b2abf5a553d99c; no voters:
I20240502 20:46:21.937453 25221 leader_election.cc:288] T 00000000000000000000000000000000 P 67ad22b014554d79b4b2abf5a553d99c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:46:21.937588 25224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67ad22b014554d79b4b2abf5a553d99c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:21.937772 25224 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67ad22b014554d79b4b2abf5a553d99c [term 1 LEADER]: Becoming Leader. State: Replica: 67ad22b014554d79b4b2abf5a553d99c, State: Running, Role: LEADER
I20240502 20:46:21.937934 25221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67ad22b014554d79b4b2abf5a553d99c [sys.catalog]: configured and running, proceeding with master startup.
I20240502 20:46:21.937902 25224 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67ad22b014554d79b4b2abf5a553d99c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ad22b014554d79b4b2abf5a553d99c" member_type: VOTER }
I20240502 20:46:21.938345 25225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67ad22b014554d79b4b2abf5a553d99c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67ad22b014554d79b4b2abf5a553d99c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ad22b014554d79b4b2abf5a553d99c" member_type: VOTER } }
I20240502 20:46:21.938454 25225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67ad22b014554d79b4b2abf5a553d99c [sys.catalog]: This master's current role is: LEADER
I20240502 20:46:21.938364 25226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67ad22b014554d79b4b2abf5a553d99c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67ad22b014554d79b4b2abf5a553d99c. Latest consensus state: current_term: 1 leader_uuid: "67ad22b014554d79b4b2abf5a553d99c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ad22b014554d79b4b2abf5a553d99c" member_type: VOTER } }
I20240502 20:46:21.938674 25226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67ad22b014554d79b4b2abf5a553d99c [sys.catalog]: This master's current role is: LEADER
I20240502 20:46:21.939026 25229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 20:46:21.939757 25229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 20:46:21.939915 22946 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 20:46:21.941514 25229 catalog_manager.cc:1347] Generated new cluster ID: 41948d37da484235aa03b46325ab363d
I20240502 20:46:21.941563 25229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 20:46:21.952211 25229 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 20:46:21.952760 25229 catalog_manager.cc:1504] Loading token signing keys...
I20240502 20:46:21.961249 25229 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 67ad22b014554d79b4b2abf5a553d99c: Generated new TSK 0
I20240502 20:46:21.961438 25229 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 20:46:21.972352 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:21.974773 25244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:21.974887 25243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:21.975540 25246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:21.975978 22946 server_base.cc:1034] running on GCE node
I20240502 20:46:21.976127 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:46:21.976164 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:46:21.976179 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682781976179 us; error 0 us; skew 500 ppm
I20240502 20:46:21.977056 22946 webserver.cc:460] Webserver started at http://127.22.104.129:43863/ using document root <none> and password file <none>
I20240502 20:46:21.977200 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:21.977246 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:21.977321 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:21.977720 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714682726530369-22946-0/minicluster-data/ts-0-root/instance:
uuid: "61820c25d42645299ee5e143f6a24122"
format_stamp: "Formatted at 2024-05-02 20:46:21 on dist-test-slave-rgns"
I20240502 20:46:21.979125 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 20:46:21.979939 25251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:21.980146 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:21.980209 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714682726530369-22946-0/minicluster-data/ts-0-root
uuid: "61820c25d42645299ee5e143f6a24122"
format_stamp: "Formatted at 2024-05-02 20:46:21 on dist-test-slave-rgns"
I20240502 20:46:21.980278 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714682726530369-22946-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714682726530369-22946-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714682726530369-22946-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:21.995818 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:21.996212 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:21.996497 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:21.996970 22946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:21.997009 22946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:21.997052 22946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:21.997081 22946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:22.000998 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.129:36305
I20240502 20:46:22.001032 25317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.129:36305 every 8 connection(s)
I20240502 20:46:22.001776 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:22.003590 25322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:22.004029 25323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:22.004727 25325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:22.005345 22946 server_base.cc:1034] running on GCE node
I20240502 20:46:22.005539 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:46:22.005609 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:46:22.005631 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682782005630 us; error 0 us; skew 500 ppm
I20240502 20:46:22.006167 25318 heartbeater.cc:347] Connected to a master server at 127.22.104.190:37589
I20240502 20:46:22.006260 25318 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:22.006501 25318 heartbeater.cc:511] Master 127.22.104.190:37589 requested a full tablet report, sending...
I20240502 20:46:22.006676 22946 webserver.cc:460] Webserver started at http://127.22.104.130:46851/ using document root <none> and password file <none>
I20240502 20:46:22.006863 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:22.006929 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:22.007004 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:22.007193 25183 ts_manager.cc:194] Registered new tserver with Master: 61820c25d42645299ee5e143f6a24122 (127.22.104.129:36305)
I20240502 20:46:22.007408 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714682726530369-22946-0/minicluster-data/ts-1-root/instance:
uuid: "fbecb346db3642e6a7997076d13f8208"
format_stamp: "Formatted at 2024-05-02 20:46:22 on dist-test-slave-rgns"
I20240502 20:46:22.007948 25183 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58448
I20240502 20:46:22.008898 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 20:46:22.009745 25330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:22.009960 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:22.010027 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714682726530369-22946-0/minicluster-data/ts-1-root
uuid: "fbecb346db3642e6a7997076d13f8208"
format_stamp: "Formatted at 2024-05-02 20:46:22 on dist-test-slave-rgns"
I20240502 20:46:22.010097 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714682726530369-22946-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714682726530369-22946-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714682726530369-22946-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:22.030865 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:22.031320 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:22.031625 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:22.032099 22946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:22.032140 22946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:22.032184 22946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:22.032212 22946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:22.036339 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.130:44933
I20240502 20:46:22.036394 25393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.130:44933 every 8 connection(s)
I20240502 20:46:22.037111 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:22.039080 25397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:22.039533 25398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:22.040040 25400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:22.040678 22946 server_base.cc:1034] running on GCE node
I20240502 20:46:22.040918 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:46:22.040973 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:46:22.040993 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682782040993 us; error 0 us; skew 500 ppm
I20240502 20:46:22.041146 25394 heartbeater.cc:347] Connected to a master server at 127.22.104.190:37589
I20240502 20:46:22.041229 25394 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:22.041445 25394 heartbeater.cc:511] Master 127.22.104.190:37589 requested a full tablet report, sending...
I20240502 20:46:22.041925 22946 webserver.cc:460] Webserver started at http://127.22.104.131:43351/ using document root <none> and password file <none>
I20240502 20:46:22.042090 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:22.042109 25183 ts_manager.cc:194] Registered new tserver with Master: fbecb346db3642e6a7997076d13f8208 (127.22.104.130:44933)
I20240502 20:46:22.042150 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:22.042230 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:22.042675 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714682726530369-22946-0/minicluster-data/ts-2-root/instance:
uuid: "7df056f18b50443183c41f4bc609d46c"
format_stamp: "Formatted at 2024-05-02 20:46:22 on dist-test-slave-rgns"
I20240502 20:46:22.042774 25183 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58452
I20240502 20:46:22.044209 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:22.045035 25406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:22.045293 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:22.045356 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714682726530369-22946-0/minicluster-data/ts-2-root
uuid: "7df056f18b50443183c41f4bc609d46c"
format_stamp: "Formatted at 2024-05-02 20:46:22 on dist-test-slave-rgns"
I20240502 20:46:22.045411 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714682726530369-22946-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714682726530369-22946-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714682726530369-22946-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:22.053906 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:22.054252 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:22.054513 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:22.054997 22946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:22.055035 22946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:22.055069 22946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:22.055095 22946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:22.059446 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.131:46821
I20240502 20:46:22.059499 25472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.131:46821 every 8 connection(s)
I20240502 20:46:22.063545 25473 heartbeater.cc:347] Connected to a master server at 127.22.104.190:37589
I20240502 20:46:22.063648 25473 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:22.063861 25473 heartbeater.cc:511] Master 127.22.104.190:37589 requested a full tablet report, sending...
I20240502 20:46:22.064460 25183 ts_manager.cc:194] Registered new tserver with Master: 7df056f18b50443183c41f4bc609d46c (127.22.104.131:46821)
I20240502 20:46:22.064963 25183 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58456
I20240502 20:46:22.065407 22946 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005571164s
I20240502 20:46:23.008903 25318 heartbeater.cc:503] Master 127.22.104.190:37589 was elected leader, sending a full tablet report...
I20240502 20:46:23.043763 25394 heartbeater.cc:503] Master 127.22.104.190:37589 was elected leader, sending a full tablet report...
I20240502 20:46:23.065840 25473 heartbeater.cc:503] Master 127.22.104.190:37589 was elected leader, sending a full tablet report...
I20240502 20:46:23.939975 25239 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:46:25.940385 25239 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:46:27.940761 25239 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:46:29.941118 25239 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:46:31.941505 25239 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:46:33.941901 25239 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:46:34.142786 22946 tablet_server.cc:178] TabletServer@127.22.104.129:0 shutting down...
I20240502 20:46:34.146541 22946 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 20:46:34.158597 22946 tablet_server.cc:194] TabletServer@127.22.104.129:0 shutdown complete.
I20240502 20:46:34.160559 22946 tablet_server.cc:178] TabletServer@127.22.104.130:0 shutting down...
I20240502 20:46:34.163998 22946 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 20:46:34.175949 22946 tablet_server.cc:194] TabletServer@127.22.104.130:0 shutdown complete.
I20240502 20:46:34.177821 22946 tablet_server.cc:178] TabletServer@127.22.104.131:0 shutting down...
I20240502 20:46:34.181238 22946 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 20:46:34.193253 22946 tablet_server.cc:194] TabletServer@127.22.104.131:0 shutdown complete.
I20240502 20:46:34.195206 22946 master.cc:537] Master@127.22.104.190:37589 shutting down...
I20240502 20:46:34.197957 22946 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 67ad22b014554d79b4b2abf5a553d99c [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:34.198134 22946 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 67ad22b014554d79b4b2abf5a553d99c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:34.198202 22946 tablet_replica.cc:330] T 00000000000000000000000000000000 P 67ad22b014554d79b4b2abf5a553d99c: stopping tablet replica
I20240502 20:46:34.210388 22946 master.cc:552] Master@127.22.104.190:37589 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12311 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20240502 20:46:34.215685 22946 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.22.104.190:39787
I20240502 20:46:34.216012 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:34.217907 25477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:34.218396 25480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:34.218931 22946 server_base.cc:1034] running on GCE node
W20240502 20:46:34.219583 25478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:34.219785 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:46:34.219828 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:46:34.219843 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682794219843 us; error 0 us; skew 500 ppm
I20240502 20:46:34.220693 22946 webserver.cc:460] Webserver started at http://127.22.104.190:46653/ using document root <none> and password file <none>
I20240502 20:46:34.220834 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:34.220875 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:34.220932 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:34.221294 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714682726530369-22946-0/minicluster-data/master-0-root/instance:
uuid: "3b046bc2b77a4990b97bd12863d03e8f"
format_stamp: "Formatted at 2024-05-02 20:46:34 on dist-test-slave-rgns"
I20240502 20:46:34.222689 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:34.223623 25485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:34.223858 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:34.223919 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714682726530369-22946-0/minicluster-data/master-0-root
uuid: "3b046bc2b77a4990b97bd12863d03e8f"
format_stamp: "Formatted at 2024-05-02 20:46:34 on dist-test-slave-rgns"
I20240502 20:46:34.223987 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714682726530369-22946-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714682726530369-22946-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714682726530369-22946-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:34.234783 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:34.235164 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:34.239114 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.190:39787
I20240502 20:46:34.240384 25537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.190:39787 every 8 connection(s)
I20240502 20:46:34.240832 25538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:34.242563 25538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b046bc2b77a4990b97bd12863d03e8f: Bootstrap starting.
I20240502 20:46:34.243340 25538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b046bc2b77a4990b97bd12863d03e8f: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:34.244379 25538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b046bc2b77a4990b97bd12863d03e8f: No bootstrap required, opened a new log
I20240502 20:46:34.244742 25538 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b046bc2b77a4990b97bd12863d03e8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b046bc2b77a4990b97bd12863d03e8f" member_type: VOTER }
I20240502 20:46:34.244829 25538 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b046bc2b77a4990b97bd12863d03e8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:34.244850 25538 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b046bc2b77a4990b97bd12863d03e8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b046bc2b77a4990b97bd12863d03e8f, State: Initialized, Role: FOLLOWER
I20240502 20:46:34.244949 25538 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b046bc2b77a4990b97bd12863d03e8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b046bc2b77a4990b97bd12863d03e8f" member_type: VOTER }
I20240502 20:46:34.245000 25538 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b046bc2b77a4990b97bd12863d03e8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:46:34.245020 25538 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b046bc2b77a4990b97bd12863d03e8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:46:34.245056 25538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b046bc2b77a4990b97bd12863d03e8f [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:34.245689 25538 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b046bc2b77a4990b97bd12863d03e8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b046bc2b77a4990b97bd12863d03e8f" member_type: VOTER }
I20240502 20:46:34.245802 25538 leader_election.cc:302] T 00000000000000000000000000000000 P 3b046bc2b77a4990b97bd12863d03e8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b046bc2b77a4990b97bd12863d03e8f; no voters:
I20240502 20:46:34.245945 25538 leader_election.cc:288] T 00000000000000000000000000000000 P 3b046bc2b77a4990b97bd12863d03e8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:46:34.246083 25541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b046bc2b77a4990b97bd12863d03e8f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:34.246333 25538 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b046bc2b77a4990b97bd12863d03e8f [sys.catalog]: configured and running, proceeding with master startup.
I20240502 20:46:34.246344 25541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b046bc2b77a4990b97bd12863d03e8f [term 1 LEADER]: Becoming Leader. State: Replica: 3b046bc2b77a4990b97bd12863d03e8f, State: Running, Role: LEADER
I20240502 20:46:34.246500 25541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b046bc2b77a4990b97bd12863d03e8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b046bc2b77a4990b97bd12863d03e8f" member_type: VOTER }
I20240502 20:46:34.246951 25542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b046bc2b77a4990b97bd12863d03e8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b046bc2b77a4990b97bd12863d03e8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b046bc2b77a4990b97bd12863d03e8f" member_type: VOTER } }
I20240502 20:46:34.246991 25543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b046bc2b77a4990b97bd12863d03e8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b046bc2b77a4990b97bd12863d03e8f. Latest consensus state: current_term: 1 leader_uuid: "3b046bc2b77a4990b97bd12863d03e8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b046bc2b77a4990b97bd12863d03e8f" member_type: VOTER } }
I20240502 20:46:34.247144 25543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b046bc2b77a4990b97bd12863d03e8f [sys.catalog]: This master's current role is: LEADER
I20240502 20:46:34.247131 25542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b046bc2b77a4990b97bd12863d03e8f [sys.catalog]: This master's current role is: LEADER
I20240502 20:46:34.247690 25550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 20:46:34.248625 25550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 20:46:34.248800 22946 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 20:46:34.250394 25550 catalog_manager.cc:1347] Generated new cluster ID: 2365e888f445454e81355b7654e4ff44
I20240502 20:46:34.250440 25550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 20:46:34.255575 25550 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 20:46:34.256119 25550 catalog_manager.cc:1504] Loading token signing keys...
I20240502 20:46:34.265213 25550 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3b046bc2b77a4990b97bd12863d03e8f: Generated new TSK 0
I20240502 20:46:34.265408 25550 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 20:46:34.281232 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:34.283833 25560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:34.283931 25559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:34.284659 25563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:34.285200 22946 server_base.cc:1034] running on GCE node
I20240502 20:46:34.285362 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:46:34.285400 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:46:34.285420 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682794285420 us; error 0 us; skew 500 ppm
I20240502 20:46:34.286557 22946 webserver.cc:460] Webserver started at http://127.22.104.129:42009/ using document root <none> and password file <none>
I20240502 20:46:34.286767 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:34.286829 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:34.286896 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:34.287303 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714682726530369-22946-0/minicluster-data/ts-0-root/instance:
uuid: "f94a420df46c4841a2961f25aa6c5c7b"
format_stamp: "Formatted at 2024-05-02 20:46:34 on dist-test-slave-rgns"
I20240502 20:46:34.288728 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 20:46:34.289649 25568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:34.289875 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:34.289937 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714682726530369-22946-0/minicluster-data/ts-0-root
uuid: "f94a420df46c4841a2961f25aa6c5c7b"
format_stamp: "Formatted at 2024-05-02 20:46:34 on dist-test-slave-rgns"
I20240502 20:46:34.290016 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714682726530369-22946-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714682726530369-22946-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714682726530369-22946-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:34.301896 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:34.302325 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:34.302613 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:34.303159 22946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:34.303201 22946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:34.303246 22946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:34.303272 22946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:34.307333 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.129:45851
I20240502 20:46:34.307374 25631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.129:45851 every 8 connection(s)
I20240502 20:46:34.308115 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:34.310366 25639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:34.310478 25636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:34.310894 25637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:34.311635 22946 server_base.cc:1034] running on GCE node
I20240502 20:46:34.311834 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:46:34.311872 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:46:34.311890 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682794311890 us; error 0 us; skew 500 ppm
I20240502 20:46:34.312808 25632 heartbeater.cc:347] Connected to a master server at 127.22.104.190:39787
I20240502 20:46:34.312845 22946 webserver.cc:460] Webserver started at http://127.22.104.130:34365/ using document root <none> and password file <none>
I20240502 20:46:34.312907 25632 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:34.312996 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:34.313052 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:34.313127 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:34.313146 25632 heartbeater.cc:511] Master 127.22.104.190:39787 requested a full tablet report, sending...
I20240502 20:46:34.313580 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714682726530369-22946-0/minicluster-data/ts-1-root/instance:
uuid: "caa9dec3b0d940458e7d7d937e1d5605"
format_stamp: "Formatted at 2024-05-02 20:46:34 on dist-test-slave-rgns"
I20240502 20:46:34.313784 25502 ts_manager.cc:194] Registered new tserver with Master: f94a420df46c4841a2961f25aa6c5c7b (127.22.104.129:45851)
I20240502 20:46:34.314456 25502 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40704
I20240502 20:46:34.315093 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 20:46:34.316009 25644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:34.316247 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:34.316310 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714682726530369-22946-0/minicluster-data/ts-1-root
uuid: "caa9dec3b0d940458e7d7d937e1d5605"
format_stamp: "Formatted at 2024-05-02 20:46:34 on dist-test-slave-rgns"
I20240502 20:46:34.316377 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714682726530369-22946-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714682726530369-22946-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714682726530369-22946-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:34.320813 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:34.321148 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:34.321413 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:34.321861 22946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:34.321898 22946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:34.321938 22946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:34.321965 22946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:34.325929 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.130:33961
I20240502 20:46:34.325990 25708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.130:33961 every 8 connection(s)
I20240502 20:46:34.326655 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:34.328732 25713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:34.328918 25712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:34.329339 25715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:34.329809 22946 server_base.cc:1034] running on GCE node
I20240502 20:46:34.329998 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:46:34.330039 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:46:34.330058 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682794330057 us; error 0 us; skew 500 ppm
I20240502 20:46:34.330564 25709 heartbeater.cc:347] Connected to a master server at 127.22.104.190:39787
I20240502 20:46:34.330667 25709 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:34.330902 25709 heartbeater.cc:511] Master 127.22.104.190:39787 requested a full tablet report, sending...
I20240502 20:46:34.330997 22946 webserver.cc:460] Webserver started at http://127.22.104.131:34075/ using document root <none> and password file <none>
I20240502 20:46:34.331146 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:34.331202 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:34.331308 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:34.331555 25502 ts_manager.cc:194] Registered new tserver with Master: caa9dec3b0d940458e7d7d937e1d5605 (127.22.104.130:33961)
I20240502 20:46:34.331737 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714682726530369-22946-0/minicluster-data/ts-2-root/instance:
uuid: "6d2038362af745adb817e6c63f417b71"
format_stamp: "Formatted at 2024-05-02 20:46:34 on dist-test-slave-rgns"
I20240502 20:46:34.332216 25502 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40720
I20240502 20:46:34.333271 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 20:46:34.334127 25720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:34.334391 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:34.334453 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714682726530369-22946-0/minicluster-data/ts-2-root
uuid: "6d2038362af745adb817e6c63f417b71"
format_stamp: "Formatted at 2024-05-02 20:46:34 on dist-test-slave-rgns"
I20240502 20:46:34.334520 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714682726530369-22946-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714682726530369-22946-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714682726530369-22946-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:34.347745 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:34.348179 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:34.348500 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:34.348989 22946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:34.349030 22946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:34.349061 22946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:34.349074 22946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:34.353586 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.131:44327
I20240502 20:46:34.353689 25783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.131:44327 every 8 connection(s)
I20240502 20:46:34.354337 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:34.356177 25787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:34.356459 25788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:34.356856 25790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:34.357605 22946 server_base.cc:1034] running on GCE node
I20240502 20:46:34.357787 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:46:34.357828 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:46:34.357843 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682794357843 us; error 0 us; skew 500 ppm
I20240502 20:46:34.358846 22946 webserver.cc:460] Webserver started at http://127.22.104.132:39589/ using document root <none> and password file <none>
I20240502 20:46:34.358922 25784 heartbeater.cc:347] Connected to a master server at 127.22.104.190:39787
I20240502 20:46:34.359026 25784 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:34.359123 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:34.359187 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:34.359261 25784 heartbeater.cc:511] Master 127.22.104.190:39787 requested a full tablet report, sending...
I20240502 20:46:34.359263 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:34.359774 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714682726530369-22946-0/minicluster-data/ts-3-root/instance:
uuid: "2d2920d829614cb88ad4fbcfa6585a9e"
format_stamp: "Formatted at 2024-05-02 20:46:34 on dist-test-slave-rgns"
I20240502 20:46:34.359974 25502 ts_manager.cc:194] Registered new tserver with Master: 6d2038362af745adb817e6c63f417b71 (127.22.104.131:44327)
I20240502 20:46:34.360497 25502 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40730
I20240502 20:46:34.361299 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:34.362151 25796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:34.362416 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:34.362484 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714682726530369-22946-0/minicluster-data/ts-3-root
uuid: "2d2920d829614cb88ad4fbcfa6585a9e"
format_stamp: "Formatted at 2024-05-02 20:46:34 on dist-test-slave-rgns"
I20240502 20:46:34.362569 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714682726530369-22946-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714682726530369-22946-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714682726530369-22946-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:34.375571 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:34.376026 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:34.376336 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:34.376856 22946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:34.376897 22946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:34.376942 22946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:34.376971 22946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:34.381436 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.132:41733
I20240502 20:46:34.381505 25859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.132:41733 every 8 connection(s)
I20240502 20:46:34.385747 25860 heartbeater.cc:347] Connected to a master server at 127.22.104.190:39787
I20240502 20:46:34.385855 25860 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:34.386096 25860 heartbeater.cc:511] Master 127.22.104.190:39787 requested a full tablet report, sending...
I20240502 20:46:34.386730 25502 ts_manager.cc:194] Registered new tserver with Master: 2d2920d829614cb88ad4fbcfa6585a9e (127.22.104.132:41733)
I20240502 20:46:34.387254 25502 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40744
I20240502 20:46:34.387495 22946 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.00565057s
I20240502 20:46:35.315438 25632 heartbeater.cc:503] Master 127.22.104.190:39787 was elected leader, sending a full tablet report...
I20240502 20:46:35.333112 25709 heartbeater.cc:503] Master 127.22.104.190:39787 was elected leader, sending a full tablet report...
I20240502 20:46:35.361500 25784 heartbeater.cc:503] Master 127.22.104.190:39787 was elected leader, sending a full tablet report...
I20240502 20:46:35.388242 25860 heartbeater.cc:503] Master 127.22.104.190:39787 was elected leader, sending a full tablet report...
I20240502 20:46:35.419010 22946 test_util.cc:268] Using random seed: -1293130370
I20240502 20:46:35.427567 25502 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40764:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 20:46:35.438473 25672 tablet_service.cc:1467] Processing CreateTablet for tablet 5561edae61f74ed697288bd1d257e1ee (DEFAULT_TABLE table=test-workload [id=86cac000c2d34d38999e2103e7507ca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:35.438602 25824 tablet_service.cc:1467] Processing CreateTablet for tablet 5561edae61f74ed697288bd1d257e1ee (DEFAULT_TABLE table=test-workload [id=86cac000c2d34d38999e2103e7507ca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:35.438684 25596 tablet_service.cc:1467] Processing CreateTablet for tablet 5561edae61f74ed697288bd1d257e1ee (DEFAULT_TABLE table=test-workload [id=86cac000c2d34d38999e2103e7507ca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:35.438843 25672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5561edae61f74ed697288bd1d257e1ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:35.438889 25824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5561edae61f74ed697288bd1d257e1ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:35.438910 25596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5561edae61f74ed697288bd1d257e1ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:35.441097 25881 tablet_bootstrap.cc:492] T 5561edae61f74ed697288bd1d257e1ee P 2d2920d829614cb88ad4fbcfa6585a9e: Bootstrap starting.
I20240502 20:46:35.441542 25883 tablet_bootstrap.cc:492] T 5561edae61f74ed697288bd1d257e1ee P caa9dec3b0d940458e7d7d937e1d5605: Bootstrap starting.
I20240502 20:46:35.441570 25882 tablet_bootstrap.cc:492] T 5561edae61f74ed697288bd1d257e1ee P f94a420df46c4841a2961f25aa6c5c7b: Bootstrap starting.
I20240502 20:46:35.442034 25881 tablet_bootstrap.cc:654] T 5561edae61f74ed697288bd1d257e1ee P 2d2920d829614cb88ad4fbcfa6585a9e: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:35.442467 25883 tablet_bootstrap.cc:654] T 5561edae61f74ed697288bd1d257e1ee P caa9dec3b0d940458e7d7d937e1d5605: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:35.442467 25882 tablet_bootstrap.cc:654] T 5561edae61f74ed697288bd1d257e1ee P f94a420df46c4841a2961f25aa6c5c7b: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:35.443113 25881 tablet_bootstrap.cc:492] T 5561edae61f74ed697288bd1d257e1ee P 2d2920d829614cb88ad4fbcfa6585a9e: No bootstrap required, opened a new log
I20240502 20:46:35.443212 25881 ts_tablet_manager.cc:1367] T 5561edae61f74ed697288bd1d257e1ee P 2d2920d829614cb88ad4fbcfa6585a9e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:35.443683 25882 tablet_bootstrap.cc:492] T 5561edae61f74ed697288bd1d257e1ee P f94a420df46c4841a2961f25aa6c5c7b: No bootstrap required, opened a new log
I20240502 20:46:35.443666 25881 raft_consensus.cc:348] T 5561edae61f74ed697288bd1d257e1ee P 2d2920d829614cb88ad4fbcfa6585a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa9dec3b0d940458e7d7d937e1d5605" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 33961 } } peers { permanent_uuid: "2d2920d829614cb88ad4fbcfa6585a9e" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 41733 } } peers { permanent_uuid: "f94a420df46c4841a2961f25aa6c5c7b" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 45851 } }
I20240502 20:46:35.443789 25882 ts_tablet_manager.cc:1367] T 5561edae61f74ed697288bd1d257e1ee P f94a420df46c4841a2961f25aa6c5c7b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 20:46:35.443800 25883 tablet_bootstrap.cc:492] T 5561edae61f74ed697288bd1d257e1ee P caa9dec3b0d940458e7d7d937e1d5605: No bootstrap required, opened a new log
I20240502 20:46:35.443903 25881 raft_consensus.cc:374] T 5561edae61f74ed697288bd1d257e1ee P 2d2920d829614cb88ad4fbcfa6585a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:35.444054 25883 ts_tablet_manager.cc:1367] T 5561edae61f74ed697288bd1d257e1ee P caa9dec3b0d940458e7d7d937e1d5605: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:35.444073 25881 raft_consensus.cc:729] T 5561edae61f74ed697288bd1d257e1ee P 2d2920d829614cb88ad4fbcfa6585a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d2920d829614cb88ad4fbcfa6585a9e, State: Initialized, Role: FOLLOWER
I20240502 20:46:35.444269 25881 consensus_queue.cc:260] T 5561edae61f74ed697288bd1d257e1ee P 2d2920d829614cb88ad4fbcfa6585a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa9dec3b0d940458e7d7d937e1d5605" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 33961 } } peers { permanent_uuid: "2d2920d829614cb88ad4fbcfa6585a9e" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 41733 } } peers { permanent_uuid: "f94a420df46c4841a2961f25aa6c5c7b" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 45851 } }
I20240502 20:46:35.444365 25882 raft_consensus.cc:348] T 5561edae61f74ed697288bd1d257e1ee P f94a420df46c4841a2961f25aa6c5c7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa9dec3b0d940458e7d7d937e1d5605" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 33961 } } peers { permanent_uuid: "2d2920d829614cb88ad4fbcfa6585a9e" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 41733 } } peers { permanent_uuid: "f94a420df46c4841a2961f25aa6c5c7b" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 45851 } }
I20240502 20:46:35.444406 25883 raft_consensus.cc:348] T 5561edae61f74ed697288bd1d257e1ee P caa9dec3b0d940458e7d7d937e1d5605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa9dec3b0d940458e7d7d937e1d5605" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 33961 } } peers { permanent_uuid: "2d2920d829614cb88ad4fbcfa6585a9e" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 41733 } } peers { permanent_uuid: "f94a420df46c4841a2961f25aa6c5c7b" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 45851 } }
I20240502 20:46:35.444491 25882 raft_consensus.cc:374] T 5561edae61f74ed697288bd1d257e1ee P f94a420df46c4841a2961f25aa6c5c7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:35.444514 25883 raft_consensus.cc:374] T 5561edae61f74ed697288bd1d257e1ee P caa9dec3b0d940458e7d7d937e1d5605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:35.444535 25882 raft_consensus.cc:729] T 5561edae61f74ed697288bd1d257e1ee P f94a420df46c4841a2961f25aa6c5c7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f94a420df46c4841a2961f25aa6c5c7b, State: Initialized, Role: FOLLOWER
I20240502 20:46:35.444542 25883 raft_consensus.cc:729] T 5561edae61f74ed697288bd1d257e1ee P caa9dec3b0d940458e7d7d937e1d5605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caa9dec3b0d940458e7d7d937e1d5605, State: Initialized, Role: FOLLOWER
I20240502 20:46:35.444720 25883 consensus_queue.cc:260] T 5561edae61f74ed697288bd1d257e1ee P caa9dec3b0d940458e7d7d937e1d5605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa9dec3b0d940458e7d7d937e1d5605" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 33961 } } peers { permanent_uuid: "2d2920d829614cb88ad4fbcfa6585a9e" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 41733 } } peers { permanent_uuid: "f94a420df46c4841a2961f25aa6c5c7b" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 45851 } }
I20240502 20:46:35.444689 25882 consensus_queue.cc:260] T 5561edae61f74ed697288bd1d257e1ee P f94a420df46c4841a2961f25aa6c5c7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa9dec3b0d940458e7d7d937e1d5605" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 33961 } } peers { permanent_uuid: "2d2920d829614cb88ad4fbcfa6585a9e" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 41733 } } peers { permanent_uuid: "f94a420df46c4841a2961f25aa6c5c7b" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 45851 } }
I20240502 20:46:35.445040 25883 ts_tablet_manager.cc:1398] T 5561edae61f74ed697288bd1d257e1ee P caa9dec3b0d940458e7d7d937e1d5605: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:35.445101 25882 ts_tablet_manager.cc:1398] T 5561edae61f74ed697288bd1d257e1ee P f94a420df46c4841a2961f25aa6c5c7b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:35.444697 25881 ts_tablet_manager.cc:1398] T 5561edae61f74ed697288bd1d257e1ee P 2d2920d829614cb88ad4fbcfa6585a9e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 20:46:35.477538 25888 raft_consensus.cc:482] T 5561edae61f74ed697288bd1d257e1ee P caa9dec3b0d940458e7d7d937e1d5605 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:35.477694 25888 raft_consensus.cc:504] T 5561edae61f74ed697288bd1d257e1ee P caa9dec3b0d940458e7d7d937e1d5605 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa9dec3b0d940458e7d7d937e1d5605" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 33961 } } peers { permanent_uuid: "2d2920d829614cb88ad4fbcfa6585a9e" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 41733 } } peers { permanent_uuid: "f94a420df46c4841a2961f25aa6c5c7b" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 45851 } }
I20240502 20:46:35.478199 25888 leader_election.cc:288] T 5561edae61f74ed697288bd1d257e1ee P caa9dec3b0d940458e7d7d937e1d5605 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d2920d829614cb88ad4fbcfa6585a9e (127.22.104.132:41733), f94a420df46c4841a2961f25aa6c5c7b (127.22.104.129:45851)
I20240502 20:46:35.482945 25834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5561edae61f74ed697288bd1d257e1ee" candidate_uuid: "caa9dec3b0d940458e7d7d937e1d5605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d2920d829614cb88ad4fbcfa6585a9e" is_pre_election: true
I20240502 20:46:35.482945 25606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5561edae61f74ed697288bd1d257e1ee" candidate_uuid: "caa9dec3b0d940458e7d7d937e1d5605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f94a420df46c4841a2961f25aa6c5c7b" is_pre_election: true
I20240502 20:46:35.483179 25606 raft_consensus.cc:2454] T 5561edae61f74ed697288bd1d257e1ee P f94a420df46c4841a2961f25aa6c5c7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate caa9dec3b0d940458e7d7d937e1d5605 in term 0.
I20240502 20:46:35.483179 25834 raft_consensus.cc:2454] T 5561edae61f74ed697288bd1d257e1ee P 2d2920d829614cb88ad4fbcfa6585a9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate caa9dec3b0d940458e7d7d937e1d5605 in term 0.
I20240502 20:46:35.483601 25648 leader_election.cc:302] T 5561edae61f74ed697288bd1d257e1ee P caa9dec3b0d940458e7d7d937e1d5605 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2d2920d829614cb88ad4fbcfa6585a9e, caa9dec3b0d940458e7d7d937e1d5605, f94a420df46c4841a2961f25aa6c5c7b; no voters:
I20240502 20:46:35.483805 25888 raft_consensus.cc:2781] T 5561edae61f74ed697288bd1d257e1ee P caa9dec3b0d940458e7d7d937e1d5605 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:35.483875 25888 raft_consensus.cc:482] T 5561edae61f74ed697288bd1d257e1ee P caa9dec3b0d940458e7d7d937e1d5605 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:35.483913 25888 raft_consensus.cc:3037] T 5561edae61f74ed697288bd1d257e1ee P caa9dec3b0d940458e7d7d937e1d5605 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:35.484769 25888 raft_consensus.cc:504] T 5561edae61f74ed697288bd1d257e1ee P caa9dec3b0d940458e7d7d937e1d5605 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa9dec3b0d940458e7d7d937e1d5605" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 33961 } } peers { permanent_uuid: "2d2920d829614cb88ad4fbcfa6585a9e" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 41733 } } peers { permanent_uuid: "f94a420df46c4841a2961f25aa6c5c7b" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 45851 } }
I20240502 20:46:35.485121 25888 leader_election.cc:288] T 5561edae61f74ed697288bd1d257e1ee P caa9dec3b0d940458e7d7d937e1d5605 [CANDIDATE]: Term 1 election: Requested vote from peers 2d2920d829614cb88ad4fbcfa6585a9e (127.22.104.132:41733), f94a420df46c4841a2961f25aa6c5c7b (127.22.104.129:45851)
I20240502 20:46:35.485441 25606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5561edae61f74ed697288bd1d257e1ee" candidate_uuid: "caa9dec3b0d940458e7d7d937e1d5605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f94a420df46c4841a2961f25aa6c5c7b"
I20240502 20:46:35.485420 25834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5561edae61f74ed697288bd1d257e1ee" candidate_uuid: "caa9dec3b0d940458e7d7d937e1d5605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d2920d829614cb88ad4fbcfa6585a9e"
I20240502 20:46:35.485555 25606 raft_consensus.cc:3037] T 5561edae61f74ed697288bd1d257e1ee P f94a420df46c4841a2961f25aa6c5c7b [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:35.485586 25834 raft_consensus.cc:3037] T 5561edae61f74ed697288bd1d257e1ee P 2d2920d829614cb88ad4fbcfa6585a9e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:35.486534 25606 raft_consensus.cc:2454] T 5561edae61f74ed697288bd1d257e1ee P f94a420df46c4841a2961f25aa6c5c7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate caa9dec3b0d940458e7d7d937e1d5605 in term 1.
I20240502 20:46:35.486534 25834 raft_consensus.cc:2454] T 5561edae61f74ed697288bd1d257e1ee P 2d2920d829614cb88ad4fbcfa6585a9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate caa9dec3b0d940458e7d7d937e1d5605 in term 1.
I20240502 20:46:35.486889 25647 leader_election.cc:302] T 5561edae61f74ed697288bd1d257e1ee P caa9dec3b0d940458e7d7d937e1d5605 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: caa9dec3b0d940458e7d7d937e1d5605, f94a420df46c4841a2961f25aa6c5c7b; no voters:
I20240502 20:46:35.487068 25888 raft_consensus.cc:2781] T 5561edae61f74ed697288bd1d257e1ee P caa9dec3b0d940458e7d7d937e1d5605 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:35.487269 25888 raft_consensus.cc:686] T 5561edae61f74ed697288bd1d257e1ee P caa9dec3b0d940458e7d7d937e1d5605 [term 1 LEADER]: Becoming Leader. State: Replica: caa9dec3b0d940458e7d7d937e1d5605, State: Running, Role: LEADER
I20240502 20:46:35.487422 25888 consensus_queue.cc:237] T 5561edae61f74ed697288bd1d257e1ee P caa9dec3b0d940458e7d7d937e1d5605 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa9dec3b0d940458e7d7d937e1d5605" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 33961 } } peers { permanent_uuid: "2d2920d829614cb88ad4fbcfa6585a9e" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 41733 } } peers { permanent_uuid: "f94a420df46c4841a2961f25aa6c5c7b" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 45851 } }
I20240502 20:46:35.488997 25501 catalog_manager.cc:5484] T 5561edae61f74ed697288bd1d257e1ee P caa9dec3b0d940458e7d7d937e1d5605 reported cstate change: term changed from 0 to 1, leader changed from <none> to caa9dec3b0d940458e7d7d937e1d5605 (127.22.104.130). New cstate: current_term: 1 leader_uuid: "caa9dec3b0d940458e7d7d937e1d5605" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa9dec3b0d940458e7d7d937e1d5605" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 33961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d2920d829614cb88ad4fbcfa6585a9e" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 41733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f94a420df46c4841a2961f25aa6c5c7b" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 45851 } health_report { overall_health: UNKNOWN } } }
W20240502 20:46:35.559381 25633 tablet.cc:2369] T 5561edae61f74ed697288bd1d257e1ee P f94a420df46c4841a2961f25aa6c5c7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:46:35.964538 25888 consensus_queue.cc:1035] T 5561edae61f74ed697288bd1d257e1ee P caa9dec3b0d940458e7d7d937e1d5605 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d2920d829614cb88ad4fbcfa6585a9e" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 41733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:35.967643 25897 consensus_queue.cc:1035] T 5561edae61f74ed697288bd1d257e1ee P caa9dec3b0d940458e7d7d937e1d5605 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f94a420df46c4841a2961f25aa6c5c7b" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 45851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:36.248509 25556 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:46:36.248818 25556 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 20:46:36.248885 25556 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:46:38.249131 25556 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:46:38.249449 25556 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 20:46:38.249516 25556 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:46:40.249819 25556 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:46:40.250171 25556 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 20:46:40.250243 25556 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:46:42.250572 25556 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:46:42.250924 25556 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 20:46:42.251009 25556 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:46:44.251297 25556 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:46:44.251621 25556 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 20:46:44.251689 25556 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:46:46.252085 25556 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:46:46.252420 25556 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 20:46:46.252519 25556 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:46:46.547901 22946 tablet_server.cc:178] TabletServer@127.22.104.129:0 shutting down...
I20240502 20:46:46.551246 22946 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 20:46:46.551471 22946 tablet_replica.cc:330] T 5561edae61f74ed697288bd1d257e1ee P f94a420df46c4841a2961f25aa6c5c7b: stopping tablet replica
I20240502 20:46:46.551597 22946 raft_consensus.cc:2229] T 5561edae61f74ed697288bd1d257e1ee P f94a420df46c4841a2961f25aa6c5c7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:46.551733 22946 raft_consensus.cc:2258] T 5561edae61f74ed697288bd1d257e1ee P f94a420df46c4841a2961f25aa6c5c7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:46.564484 22946 tablet_server.cc:194] TabletServer@127.22.104.129:0 shutdown complete.
I20240502 20:46:46.566610 22946 tablet_server.cc:178] TabletServer@127.22.104.130:0 shutting down...
I20240502 20:46:46.570086 22946 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 20:46:46.570268 22946 tablet_replica.cc:330] T 5561edae61f74ed697288bd1d257e1ee P caa9dec3b0d940458e7d7d937e1d5605: stopping tablet replica
I20240502 20:46:46.570381 22946 raft_consensus.cc:2229] T 5561edae61f74ed697288bd1d257e1ee P caa9dec3b0d940458e7d7d937e1d5605 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:46.570603 22946 raft_consensus.cc:2258] T 5561edae61f74ed697288bd1d257e1ee P caa9dec3b0d940458e7d7d937e1d5605 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:46.583163 22946 tablet_server.cc:194] TabletServer@127.22.104.130:0 shutdown complete.
I20240502 20:46:46.585104 22946 tablet_server.cc:178] TabletServer@127.22.104.131:0 shutting down...
I20240502 20:46:46.588285 22946 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 20:46:46.601439 22946 tablet_server.cc:194] TabletServer@127.22.104.131:0 shutdown complete.
I20240502 20:46:46.603446 22946 tablet_server.cc:178] TabletServer@127.22.104.132:0 shutting down...
I20240502 20:46:46.607014 22946 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 20:46:46.607206 22946 tablet_replica.cc:330] T 5561edae61f74ed697288bd1d257e1ee P 2d2920d829614cb88ad4fbcfa6585a9e: stopping tablet replica
I20240502 20:46:46.607334 22946 raft_consensus.cc:2229] T 5561edae61f74ed697288bd1d257e1ee P 2d2920d829614cb88ad4fbcfa6585a9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:46.607448 22946 raft_consensus.cc:2258] T 5561edae61f74ed697288bd1d257e1ee P 2d2920d829614cb88ad4fbcfa6585a9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:46.619958 22946 tablet_server.cc:194] TabletServer@127.22.104.132:0 shutdown complete.
I20240502 20:46:46.622021 22946 master.cc:537] Master@127.22.104.190:39787 shutting down...
I20240502 20:46:46.625350 22946 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3b046bc2b77a4990b97bd12863d03e8f [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:46.625543 22946 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3b046bc2b77a4990b97bd12863d03e8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:46.625604 22946 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3b046bc2b77a4990b97bd12863d03e8f: stopping tablet replica
I20240502 20:46:46.637836 22946 master.cc:552] Master@127.22.104.190:39787 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12428 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20240502 20:46:46.644316 22946 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.22.104.190:33225
I20240502 20:46:46.644677 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:46.646589 25903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:46.647272 22946 server_base.cc:1034] running on GCE node
W20240502 20:46:46.647706 25906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:46.647866 25904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:46.648058 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:46:46.648103 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:46:46.648123 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682806648122 us; error 0 us; skew 500 ppm
I20240502 20:46:46.654683 22946 webserver.cc:460] Webserver started at http://127.22.104.190:43205/ using document root <none> and password file <none>
I20240502 20:46:46.654878 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:46.654929 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:46.655009 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:46.655393 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714682726530369-22946-0/minicluster-data/master-0-root/instance:
uuid: "94591722856a4c1d9a77e6de58cc13f4"
format_stamp: "Formatted at 2024-05-02 20:46:46 on dist-test-slave-rgns"
I20240502 20:46:46.656803 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 20:46:46.657740 25911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:46.658023 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:46.658093 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714682726530369-22946-0/minicluster-data/master-0-root
uuid: "94591722856a4c1d9a77e6de58cc13f4"
format_stamp: "Formatted at 2024-05-02 20:46:46 on dist-test-slave-rgns"
I20240502 20:46:46.658154 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714682726530369-22946-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714682726530369-22946-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714682726530369-22946-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:46.665961 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:46.666301 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:46.670122 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.190:33225
I20240502 20:46:46.671937 25964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.190:33225 every 8 connection(s)
I20240502 20:46:46.672355 25965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:46.674182 25965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94591722856a4c1d9a77e6de58cc13f4: Bootstrap starting.
I20240502 20:46:46.675001 25965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94591722856a4c1d9a77e6de58cc13f4: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:46.676012 25965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94591722856a4c1d9a77e6de58cc13f4: No bootstrap required, opened a new log
I20240502 20:46:46.676386 25965 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94591722856a4c1d9a77e6de58cc13f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94591722856a4c1d9a77e6de58cc13f4" member_type: VOTER }
I20240502 20:46:46.676483 25965 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94591722856a4c1d9a77e6de58cc13f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:46.676504 25965 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94591722856a4c1d9a77e6de58cc13f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94591722856a4c1d9a77e6de58cc13f4, State: Initialized, Role: FOLLOWER
I20240502 20:46:46.676656 25965 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94591722856a4c1d9a77e6de58cc13f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94591722856a4c1d9a77e6de58cc13f4" member_type: VOTER }
I20240502 20:46:46.676734 25965 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94591722856a4c1d9a77e6de58cc13f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:46:46.676754 25965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94591722856a4c1d9a77e6de58cc13f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:46:46.676800 25965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94591722856a4c1d9a77e6de58cc13f4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:46.677454 25965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94591722856a4c1d9a77e6de58cc13f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94591722856a4c1d9a77e6de58cc13f4" member_type: VOTER }
I20240502 20:46:46.677575 25965 leader_election.cc:302] T 00000000000000000000000000000000 P 94591722856a4c1d9a77e6de58cc13f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94591722856a4c1d9a77e6de58cc13f4; no voters:
I20240502 20:46:46.677762 25965 leader_election.cc:288] T 00000000000000000000000000000000 P 94591722856a4c1d9a77e6de58cc13f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:46:46.677899 25968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94591722856a4c1d9a77e6de58cc13f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:46.678097 25968 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94591722856a4c1d9a77e6de58cc13f4 [term 1 LEADER]: Becoming Leader. State: Replica: 94591722856a4c1d9a77e6de58cc13f4, State: Running, Role: LEADER
I20240502 20:46:46.678225 25965 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94591722856a4c1d9a77e6de58cc13f4 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 20:46:46.678225 25968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94591722856a4c1d9a77e6de58cc13f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94591722856a4c1d9a77e6de58cc13f4" member_type: VOTER }
I20240502 20:46:46.678723 25969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94591722856a4c1d9a77e6de58cc13f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94591722856a4c1d9a77e6de58cc13f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94591722856a4c1d9a77e6de58cc13f4" member_type: VOTER } }
I20240502 20:46:46.678759 25970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94591722856a4c1d9a77e6de58cc13f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94591722856a4c1d9a77e6de58cc13f4. Latest consensus state: current_term: 1 leader_uuid: "94591722856a4c1d9a77e6de58cc13f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94591722856a4c1d9a77e6de58cc13f4" member_type: VOTER } }
I20240502 20:46:46.678859 25970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94591722856a4c1d9a77e6de58cc13f4 [sys.catalog]: This master's current role is: LEADER
I20240502 20:46:46.678815 25969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94591722856a4c1d9a77e6de58cc13f4 [sys.catalog]: This master's current role is: LEADER
I20240502 20:46:46.679945 22946 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
W20240502 20:46:46.680397 25984 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 94591722856a4c1d9a77e6de58cc13f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 20:46:46.680464 25984 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 20:46:46.680554 25974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 20:46:46.681169 25974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 20:46:46.682920 25974 catalog_manager.cc:1347] Generated new cluster ID: 7e9abfa62f4247b6aa8c67b598ae5088
I20240502 20:46:46.682982 25974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 20:46:46.699156 25974 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 20:46:46.699761 25974 catalog_manager.cc:1504] Loading token signing keys...
I20240502 20:46:46.704764 25974 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 94591722856a4c1d9a77e6de58cc13f4: Generated new TSK 0
I20240502 20:46:46.704942 25974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 20:46:46.712271 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:46.714279 25986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:46.714524 25989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:46.715165 25987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:46.715657 22946 server_base.cc:1034] running on GCE node
I20240502 20:46:46.715814 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:46:46.715852 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:46:46.715867 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682806715867 us; error 0 us; skew 500 ppm
I20240502 20:46:46.716717 22946 webserver.cc:460] Webserver started at http://127.22.104.129:40641/ using document root <none> and password file <none>
I20240502 20:46:46.716851 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:46.716898 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:46.716949 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:46.717294 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714682726530369-22946-0/minicluster-data/ts-0-root/instance:
uuid: "cc68dbd2972d4d3dbdd119046aff1d4d"
format_stamp: "Formatted at 2024-05-02 20:46:46 on dist-test-slave-rgns"
I20240502 20:46:46.718745 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240502 20:46:46.719575 25994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:46.719794 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:46.719857 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714682726530369-22946-0/minicluster-data/ts-0-root
uuid: "cc68dbd2972d4d3dbdd119046aff1d4d"
format_stamp: "Formatted at 2024-05-02 20:46:46 on dist-test-slave-rgns"
I20240502 20:46:46.719923 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714682726530369-22946-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714682726530369-22946-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714682726530369-22946-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:46.749433 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:46.749835 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:46.750133 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:46.750602 22946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:46.750643 22946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:46.750684 22946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:46.750741 22946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:46.754673 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.129:44605
I20240502 20:46:46.754734 26059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.129:44605 every 8 connection(s)
I20240502 20:46:46.755482 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:46.757439 26067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:46.757968 26069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:46.758569 26066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:46.759156 22946 server_base.cc:1034] running on GCE node
I20240502 20:46:46.759375 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:46:46.759424 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:46:46.759444 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682806759443 us; error 0 us; skew 500 ppm
I20240502 20:46:46.760346 26060 heartbeater.cc:347] Connected to a master server at 127.22.104.190:33225
I20240502 20:46:46.760438 26060 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:46.760491 22946 webserver.cc:460] Webserver started at http://127.22.104.130:46813/ using document root <none> and password file <none>
I20240502 20:46:46.760648 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:46.760665 26060 heartbeater.cc:511] Master 127.22.104.190:33225 requested a full tablet report, sending...
I20240502 20:46:46.760694 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:46.760793 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:46.761247 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714682726530369-22946-0/minicluster-data/ts-1-root/instance:
uuid: "4209a30894b6439fbd496e17aaff52c6"
format_stamp: "Formatted at 2024-05-02 20:46:46 on dist-test-slave-rgns"
I20240502 20:46:46.761436 25928 ts_manager.cc:194] Registered new tserver with Master: cc68dbd2972d4d3dbdd119046aff1d4d (127.22.104.129:44605)
I20240502 20:46:46.762430 25928 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49288
I20240502 20:46:46.763011 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 20:46:46.764037 26074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:46.764303 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:46.764365 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714682726530369-22946-0/minicluster-data/ts-1-root
uuid: "4209a30894b6439fbd496e17aaff52c6"
format_stamp: "Formatted at 2024-05-02 20:46:46 on dist-test-slave-rgns"
I20240502 20:46:46.764428 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714682726530369-22946-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714682726530369-22946-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714682726530369-22946-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:46.773867 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:46.774356 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:46.774755 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:46.775220 22946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:46.775259 22946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:46.775290 22946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:46.775308 22946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:46.779240 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.130:38291
I20240502 20:46:46.779320 26138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.130:38291 every 8 connection(s)
I20240502 20:46:46.780028 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:46.781917 26142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:46.782477 26145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:46.783043 22946 server_base.cc:1034] running on GCE node
W20240502 20:46:46.783416 26143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:46.783645 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:46:46.783699 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:46:46.783720 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682806783719 us; error 0 us; skew 500 ppm
I20240502 20:46:46.784076 26139 heartbeater.cc:347] Connected to a master server at 127.22.104.190:33225
I20240502 20:46:46.784178 26139 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:46.784397 26139 heartbeater.cc:511] Master 127.22.104.190:33225 requested a full tablet report, sending...
I20240502 20:46:46.784713 22946 webserver.cc:460] Webserver started at http://127.22.104.131:41311/ using document root <none> and password file <none>
I20240502 20:46:46.784876 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:46.784934 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:46.785008 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:46.785023 25928 ts_manager.cc:194] Registered new tserver with Master: 4209a30894b6439fbd496e17aaff52c6 (127.22.104.130:38291)
I20240502 20:46:46.785380 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714682726530369-22946-0/minicluster-data/ts-2-root/instance:
uuid: "df8ac45c3e75474bb9a2bd3a9944459e"
format_stamp: "Formatted at 2024-05-02 20:46:46 on dist-test-slave-rgns"
I20240502 20:46:46.785567 25928 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49304
I20240502 20:46:46.786828 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240502 20:46:46.787755 26150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:46.788066 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:46.788132 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714682726530369-22946-0/minicluster-data/ts-2-root
uuid: "df8ac45c3e75474bb9a2bd3a9944459e"
format_stamp: "Formatted at 2024-05-02 20:46:46 on dist-test-slave-rgns"
I20240502 20:46:46.788192 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714682726530369-22946-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714682726530369-22946-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714682726530369-22946-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:46.801201 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:46.801607 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:46.801882 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:46.802386 22946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:46.802428 22946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:46.802461 22946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:46.802489 22946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:46.807175 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.131:36327
I20240502 20:46:46.807235 26213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.131:36327 every 8 connection(s)
I20240502 20:46:46.812788 26214 heartbeater.cc:347] Connected to a master server at 127.22.104.190:33225
I20240502 20:46:46.812896 26214 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:46.813144 26214 heartbeater.cc:511] Master 127.22.104.190:33225 requested a full tablet report, sending...
I20240502 20:46:46.813905 25928 ts_manager.cc:194] Registered new tserver with Master: df8ac45c3e75474bb9a2bd3a9944459e (127.22.104.131:36327)
I20240502 20:46:46.814469 25928 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49308
I20240502 20:46:46.814633 22946 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.00697546s
I20240502 20:46:47.763474 26060 heartbeater.cc:503] Master 127.22.104.190:33225 was elected leader, sending a full tablet report...
I20240502 20:46:47.786425 26139 heartbeater.cc:503] Master 127.22.104.190:33225 was elected leader, sending a full tablet report...
I20240502 20:46:47.815398 26214 heartbeater.cc:503] Master 127.22.104.190:33225 was elected leader, sending a full tablet report...
I20240502 20:46:47.845314 22946 test_util.cc:268] Using random seed: -1280704065
I20240502 20:46:47.852653 25928 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49336:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 20:46:47.853094 25928 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 20:46:47.862294 26178 tablet_service.cc:1467] Processing CreateTablet for tablet d386b34cd5924368aa6400050a203565 (DEFAULT_TABLE table=test-workload [id=4cca3a6d621a415181fab96fbba4a7a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:47.862411 26103 tablet_service.cc:1467] Processing CreateTablet for tablet d386b34cd5924368aa6400050a203565 (DEFAULT_TABLE table=test-workload [id=4cca3a6d621a415181fab96fbba4a7a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:47.862597 26178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d386b34cd5924368aa6400050a203565. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:47.862655 26103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d386b34cd5924368aa6400050a203565. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:47.862597 26022 tablet_service.cc:1467] Processing CreateTablet for tablet d386b34cd5924368aa6400050a203565 (DEFAULT_TABLE table=test-workload [id=4cca3a6d621a415181fab96fbba4a7a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:47.862891 26022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d386b34cd5924368aa6400050a203565. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:47.864588 26235 tablet_bootstrap.cc:492] T d386b34cd5924368aa6400050a203565 P 4209a30894b6439fbd496e17aaff52c6: Bootstrap starting.
I20240502 20:46:47.864917 26236 tablet_bootstrap.cc:492] T d386b34cd5924368aa6400050a203565 P df8ac45c3e75474bb9a2bd3a9944459e: Bootstrap starting.
I20240502 20:46:47.865572 26235 tablet_bootstrap.cc:654] T d386b34cd5924368aa6400050a203565 P 4209a30894b6439fbd496e17aaff52c6: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:47.865762 26237 tablet_bootstrap.cc:492] T d386b34cd5924368aa6400050a203565 P cc68dbd2972d4d3dbdd119046aff1d4d: Bootstrap starting.
I20240502 20:46:47.865896 26236 tablet_bootstrap.cc:654] T d386b34cd5924368aa6400050a203565 P df8ac45c3e75474bb9a2bd3a9944459e: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:47.866593 26237 tablet_bootstrap.cc:654] T d386b34cd5924368aa6400050a203565 P cc68dbd2972d4d3dbdd119046aff1d4d: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:47.866765 26235 tablet_bootstrap.cc:492] T d386b34cd5924368aa6400050a203565 P 4209a30894b6439fbd496e17aaff52c6: No bootstrap required, opened a new log
I20240502 20:46:47.866854 26235 ts_tablet_manager.cc:1367] T d386b34cd5924368aa6400050a203565 P 4209a30894b6439fbd496e17aaff52c6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:47.866943 26236 tablet_bootstrap.cc:492] T d386b34cd5924368aa6400050a203565 P df8ac45c3e75474bb9a2bd3a9944459e: No bootstrap required, opened a new log
I20240502 20:46:47.867038 26236 ts_tablet_manager.cc:1367] T d386b34cd5924368aa6400050a203565 P df8ac45c3e75474bb9a2bd3a9944459e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:47.867283 26235 raft_consensus.cc:348] T d386b34cd5924368aa6400050a203565 P 4209a30894b6439fbd496e17aaff52c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8ac45c3e75474bb9a2bd3a9944459e" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 36327 } } peers { permanent_uuid: "4209a30894b6439fbd496e17aaff52c6" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 38291 } } peers { permanent_uuid: "cc68dbd2972d4d3dbdd119046aff1d4d" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 44605 } }
I20240502 20:46:47.867427 26235 raft_consensus.cc:374] T d386b34cd5924368aa6400050a203565 P 4209a30894b6439fbd496e17aaff52c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:47.867404 26236 raft_consensus.cc:348] T d386b34cd5924368aa6400050a203565 P df8ac45c3e75474bb9a2bd3a9944459e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8ac45c3e75474bb9a2bd3a9944459e" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 36327 } } peers { permanent_uuid: "4209a30894b6439fbd496e17aaff52c6" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 38291 } } peers { permanent_uuid: "cc68dbd2972d4d3dbdd119046aff1d4d" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 44605 } }
I20240502 20:46:47.867484 26235 raft_consensus.cc:729] T d386b34cd5924368aa6400050a203565 P 4209a30894b6439fbd496e17aaff52c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4209a30894b6439fbd496e17aaff52c6, State: Initialized, Role: FOLLOWER
I20240502 20:46:47.867497 26236 raft_consensus.cc:374] T d386b34cd5924368aa6400050a203565 P df8ac45c3e75474bb9a2bd3a9944459e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:47.867565 26236 raft_consensus.cc:729] T d386b34cd5924368aa6400050a203565 P df8ac45c3e75474bb9a2bd3a9944459e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8ac45c3e75474bb9a2bd3a9944459e, State: Initialized, Role: FOLLOWER
I20240502 20:46:47.867640 26235 consensus_queue.cc:260] T d386b34cd5924368aa6400050a203565 P 4209a30894b6439fbd496e17aaff52c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8ac45c3e75474bb9a2bd3a9944459e" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 36327 } } peers { permanent_uuid: "4209a30894b6439fbd496e17aaff52c6" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 38291 } } peers { permanent_uuid: "cc68dbd2972d4d3dbdd119046aff1d4d" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 44605 } }
I20240502 20:46:47.867780 26237 tablet_bootstrap.cc:492] T d386b34cd5924368aa6400050a203565 P cc68dbd2972d4d3dbdd119046aff1d4d: No bootstrap required, opened a new log
I20240502 20:46:47.867697 26236 consensus_queue.cc:260] T d386b34cd5924368aa6400050a203565 P df8ac45c3e75474bb9a2bd3a9944459e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8ac45c3e75474bb9a2bd3a9944459e" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 36327 } } peers { permanent_uuid: "4209a30894b6439fbd496e17aaff52c6" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 38291 } } peers { permanent_uuid: "cc68dbd2972d4d3dbdd119046aff1d4d" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 44605 } }
I20240502 20:46:47.867851 26237 ts_tablet_manager.cc:1367] T d386b34cd5924368aa6400050a203565 P cc68dbd2972d4d3dbdd119046aff1d4d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:47.868108 26235 ts_tablet_manager.cc:1398] T d386b34cd5924368aa6400050a203565 P 4209a30894b6439fbd496e17aaff52c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:47.868131 26236 ts_tablet_manager.cc:1398] T d386b34cd5924368aa6400050a203565 P df8ac45c3e75474bb9a2bd3a9944459e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:47.868882 26237 raft_consensus.cc:348] T d386b34cd5924368aa6400050a203565 P cc68dbd2972d4d3dbdd119046aff1d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8ac45c3e75474bb9a2bd3a9944459e" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 36327 } } peers { permanent_uuid: "4209a30894b6439fbd496e17aaff52c6" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 38291 } } peers { permanent_uuid: "cc68dbd2972d4d3dbdd119046aff1d4d" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 44605 } }
I20240502 20:46:47.868971 26237 raft_consensus.cc:374] T d386b34cd5924368aa6400050a203565 P cc68dbd2972d4d3dbdd119046aff1d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:47.868992 26237 raft_consensus.cc:729] T d386b34cd5924368aa6400050a203565 P cc68dbd2972d4d3dbdd119046aff1d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc68dbd2972d4d3dbdd119046aff1d4d, State: Initialized, Role: FOLLOWER
I20240502 20:46:47.869091 26237 consensus_queue.cc:260] T d386b34cd5924368aa6400050a203565 P cc68dbd2972d4d3dbdd119046aff1d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8ac45c3e75474bb9a2bd3a9944459e" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 36327 } } peers { permanent_uuid: "4209a30894b6439fbd496e17aaff52c6" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 38291 } } peers { permanent_uuid: "cc68dbd2972d4d3dbdd119046aff1d4d" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 44605 } }
I20240502 20:46:47.869505 26237 ts_tablet_manager.cc:1398] T d386b34cd5924368aa6400050a203565 P cc68dbd2972d4d3dbdd119046aff1d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:47.896353 26243 raft_consensus.cc:482] T d386b34cd5924368aa6400050a203565 P cc68dbd2972d4d3dbdd119046aff1d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:47.896521 26243 raft_consensus.cc:504] T d386b34cd5924368aa6400050a203565 P cc68dbd2972d4d3dbdd119046aff1d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8ac45c3e75474bb9a2bd3a9944459e" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 36327 } } peers { permanent_uuid: "4209a30894b6439fbd496e17aaff52c6" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 38291 } } peers { permanent_uuid: "cc68dbd2972d4d3dbdd119046aff1d4d" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 44605 } }
I20240502 20:46:47.897150 26243 leader_election.cc:288] T d386b34cd5924368aa6400050a203565 P cc68dbd2972d4d3dbdd119046aff1d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df8ac45c3e75474bb9a2bd3a9944459e (127.22.104.131:36327), 4209a30894b6439fbd496e17aaff52c6 (127.22.104.130:38291)
I20240502 20:46:47.901888 26113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d386b34cd5924368aa6400050a203565" candidate_uuid: "cc68dbd2972d4d3dbdd119046aff1d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4209a30894b6439fbd496e17aaff52c6" is_pre_election: true
I20240502 20:46:47.902053 26113 raft_consensus.cc:2454] T d386b34cd5924368aa6400050a203565 P 4209a30894b6439fbd496e17aaff52c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc68dbd2972d4d3dbdd119046aff1d4d in term 0.
I20240502 20:46:47.902455 25996 leader_election.cc:302] T d386b34cd5924368aa6400050a203565 P cc68dbd2972d4d3dbdd119046aff1d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4209a30894b6439fbd496e17aaff52c6, cc68dbd2972d4d3dbdd119046aff1d4d; no voters:
I20240502 20:46:47.902660 26243 raft_consensus.cc:2781] T d386b34cd5924368aa6400050a203565 P cc68dbd2972d4d3dbdd119046aff1d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:47.902750 26243 raft_consensus.cc:482] T d386b34cd5924368aa6400050a203565 P cc68dbd2972d4d3dbdd119046aff1d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:47.902918 26243 raft_consensus.cc:3037] T d386b34cd5924368aa6400050a203565 P cc68dbd2972d4d3dbdd119046aff1d4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:47.903541 26188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d386b34cd5924368aa6400050a203565" candidate_uuid: "cc68dbd2972d4d3dbdd119046aff1d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8ac45c3e75474bb9a2bd3a9944459e" is_pre_election: true
I20240502 20:46:47.903654 26188 raft_consensus.cc:2454] T d386b34cd5924368aa6400050a203565 P df8ac45c3e75474bb9a2bd3a9944459e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc68dbd2972d4d3dbdd119046aff1d4d in term 0.
I20240502 20:46:47.904016 26243 raft_consensus.cc:504] T d386b34cd5924368aa6400050a203565 P cc68dbd2972d4d3dbdd119046aff1d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8ac45c3e75474bb9a2bd3a9944459e" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 36327 } } peers { permanent_uuid: "4209a30894b6439fbd496e17aaff52c6" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 38291 } } peers { permanent_uuid: "cc68dbd2972d4d3dbdd119046aff1d4d" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 44605 } }
I20240502 20:46:47.904655 26243 leader_election.cc:288] T d386b34cd5924368aa6400050a203565 P cc68dbd2972d4d3dbdd119046aff1d4d [CANDIDATE]: Term 1 election: Requested vote from peers df8ac45c3e75474bb9a2bd3a9944459e (127.22.104.131:36327), 4209a30894b6439fbd496e17aaff52c6 (127.22.104.130:38291)
I20240502 20:46:47.904788 26188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d386b34cd5924368aa6400050a203565" candidate_uuid: "cc68dbd2972d4d3dbdd119046aff1d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8ac45c3e75474bb9a2bd3a9944459e"
I20240502 20:46:47.904891 26188 raft_consensus.cc:3037] T d386b34cd5924368aa6400050a203565 P df8ac45c3e75474bb9a2bd3a9944459e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:47.905586 26188 raft_consensus.cc:2454] T d386b34cd5924368aa6400050a203565 P df8ac45c3e75474bb9a2bd3a9944459e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc68dbd2972d4d3dbdd119046aff1d4d in term 1.
I20240502 20:46:47.905766 26113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d386b34cd5924368aa6400050a203565" candidate_uuid: "cc68dbd2972d4d3dbdd119046aff1d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4209a30894b6439fbd496e17aaff52c6"
I20240502 20:46:47.905845 26113 raft_consensus.cc:3037] T d386b34cd5924368aa6400050a203565 P 4209a30894b6439fbd496e17aaff52c6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:47.906507 26113 raft_consensus.cc:2454] T d386b34cd5924368aa6400050a203565 P 4209a30894b6439fbd496e17aaff52c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc68dbd2972d4d3dbdd119046aff1d4d in term 1.
I20240502 20:46:47.906775 25996 leader_election.cc:302] T d386b34cd5924368aa6400050a203565 P cc68dbd2972d4d3dbdd119046aff1d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4209a30894b6439fbd496e17aaff52c6, cc68dbd2972d4d3dbdd119046aff1d4d, df8ac45c3e75474bb9a2bd3a9944459e; no voters:
I20240502 20:46:47.907157 26243 raft_consensus.cc:2781] T d386b34cd5924368aa6400050a203565 P cc68dbd2972d4d3dbdd119046aff1d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:47.907341 26243 raft_consensus.cc:686] T d386b34cd5924368aa6400050a203565 P cc68dbd2972d4d3dbdd119046aff1d4d [term 1 LEADER]: Becoming Leader. State: Replica: cc68dbd2972d4d3dbdd119046aff1d4d, State: Running, Role: LEADER
I20240502 20:46:47.907521 26243 consensus_queue.cc:237] T d386b34cd5924368aa6400050a203565 P cc68dbd2972d4d3dbdd119046aff1d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8ac45c3e75474bb9a2bd3a9944459e" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 36327 } } peers { permanent_uuid: "4209a30894b6439fbd496e17aaff52c6" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 38291 } } peers { permanent_uuid: "cc68dbd2972d4d3dbdd119046aff1d4d" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 44605 } }
I20240502 20:46:47.909121 25928 catalog_manager.cc:5484] T d386b34cd5924368aa6400050a203565 P cc68dbd2972d4d3dbdd119046aff1d4d reported cstate change: term changed from 0 to 1, leader changed from <none> to cc68dbd2972d4d3dbdd119046aff1d4d (127.22.104.129). New cstate: current_term: 1 leader_uuid: "cc68dbd2972d4d3dbdd119046aff1d4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8ac45c3e75474bb9a2bd3a9944459e" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 36327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4209a30894b6439fbd496e17aaff52c6" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc68dbd2972d4d3dbdd119046aff1d4d" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 44605 } health_report { overall_health: HEALTHY } } }
I20240502 20:46:48.401854 26243 consensus_queue.cc:1035] T d386b34cd5924368aa6400050a203565 P cc68dbd2972d4d3dbdd119046aff1d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4209a30894b6439fbd496e17aaff52c6" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:48.404822 26243 consensus_queue.cc:1035] T d386b34cd5924368aa6400050a203565 P cc68dbd2972d4d3dbdd119046aff1d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8ac45c3e75474bb9a2bd3a9944459e" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 36327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:48.680104 25983 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:46:48.680379 25983 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 20:46:48.680439 25983 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:46:48.680544 25977 placement_policy_util.cc:407] tablet d386b34cd5924368aa6400050a203565: detected majority of replicas (2 of 3) at location L0
I20240502 20:46:49.682382 25977 placement_policy_util.cc:407] tablet d386b34cd5924368aa6400050a203565: detected majority of replicas (2 of 3) at location L0
I20240502 20:46:50.680744 25983 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:46:50.681089 25983 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 20:46:50.681155 25983 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:46:50.683985 25977 placement_policy_util.cc:407] tablet d386b34cd5924368aa6400050a203565: detected majority of replicas (2 of 3) at location L0
I20240502 20:46:51.685568 25977 placement_policy_util.cc:407] tablet d386b34cd5924368aa6400050a203565: detected majority of replicas (2 of 3) at location L0
I20240502 20:46:52.681524 25983 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:46:52.681854 25983 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 20:46:52.681921 25983 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:46:52.687137 25977 placement_policy_util.cc:407] tablet d386b34cd5924368aa6400050a203565: detected majority of replicas (2 of 3) at location L0
I20240502 20:46:53.688726 25977 placement_policy_util.cc:407] tablet d386b34cd5924368aa6400050a203565: detected majority of replicas (2 of 3) at location L0
I20240502 20:46:54.682269 25983 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:46:54.682634 25983 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 20:46:54.682725 25983 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:46:54.690359 25977 placement_policy_util.cc:407] tablet d386b34cd5924368aa6400050a203565: detected majority of replicas (2 of 3) at location L0
I20240502 20:46:55.691999 25977 placement_policy_util.cc:407] tablet d386b34cd5924368aa6400050a203565: detected majority of replicas (2 of 3) at location L0
I20240502 20:46:56.683046 25983 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:46:56.683382 25983 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 20:46:56.683450 25983 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:46:56.693668 25977 placement_policy_util.cc:407] tablet d386b34cd5924368aa6400050a203565: detected majority of replicas (2 of 3) at location L0
I20240502 20:46:57.695299 25977 placement_policy_util.cc:407] tablet d386b34cd5924368aa6400050a203565: detected majority of replicas (2 of 3) at location L0
I20240502 20:46:58.683820 25983 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:46:58.684151 25983 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 20:46:58.684219 25983 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:46:58.696933 25977 placement_policy_util.cc:407] tablet d386b34cd5924368aa6400050a203565: detected majority of replicas (2 of 3) at location L0
I20240502 20:46:58.973590 22946 tablet_server.cc:178] TabletServer@127.22.104.129:0 shutting down...
I20240502 20:46:58.977452 22946 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 20:46:58.977720 22946 tablet_replica.cc:330] T d386b34cd5924368aa6400050a203565 P cc68dbd2972d4d3dbdd119046aff1d4d: stopping tablet replica
I20240502 20:46:58.977840 22946 raft_consensus.cc:2229] T d386b34cd5924368aa6400050a203565 P cc68dbd2972d4d3dbdd119046aff1d4d [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:58.978075 22946 raft_consensus.cc:2258] T d386b34cd5924368aa6400050a203565 P cc68dbd2972d4d3dbdd119046aff1d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:58.991160 22946 tablet_server.cc:194] TabletServer@127.22.104.129:0 shutdown complete.
I20240502 20:46:58.993160 22946 tablet_server.cc:178] TabletServer@127.22.104.130:0 shutting down...
I20240502 20:46:58.996678 22946 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 20:46:58.996873 22946 tablet_replica.cc:330] T d386b34cd5924368aa6400050a203565 P 4209a30894b6439fbd496e17aaff52c6: stopping tablet replica
I20240502 20:46:58.996994 22946 raft_consensus.cc:2229] T d386b34cd5924368aa6400050a203565 P 4209a30894b6439fbd496e17aaff52c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:58.997123 22946 raft_consensus.cc:2258] T d386b34cd5924368aa6400050a203565 P 4209a30894b6439fbd496e17aaff52c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:59.009634 22946 tablet_server.cc:194] TabletServer@127.22.104.130:0 shutdown complete.
I20240502 20:46:59.011750 22946 tablet_server.cc:178] TabletServer@127.22.104.131:0 shutting down...
I20240502 20:46:59.015151 22946 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 20:46:59.015374 22946 tablet_replica.cc:330] T d386b34cd5924368aa6400050a203565 P df8ac45c3e75474bb9a2bd3a9944459e: stopping tablet replica
I20240502 20:46:59.015506 22946 raft_consensus.cc:2229] T d386b34cd5924368aa6400050a203565 P df8ac45c3e75474bb9a2bd3a9944459e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:59.015627 22946 raft_consensus.cc:2258] T d386b34cd5924368aa6400050a203565 P df8ac45c3e75474bb9a2bd3a9944459e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:59.029341 22946 tablet_server.cc:194] TabletServer@127.22.104.131:0 shutdown complete.
I20240502 20:46:59.031512 22946 master.cc:537] Master@127.22.104.190:33225 shutting down...
I20240502 20:46:59.034709 22946 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 94591722856a4c1d9a77e6de58cc13f4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:59.034905 22946 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 94591722856a4c1d9a77e6de58cc13f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:59.034971 22946 tablet_replica.cc:330] T 00000000000000000000000000000000 P 94591722856a4c1d9a77e6de58cc13f4: stopping tablet replica
I20240502 20:46:59.047269 22946 master.cc:552] Master@127.22.104.190:33225 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (12408 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
I20240502 20:46:59.053184 22946 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.22.104.190:39193
I20240502 20:46:59.053527 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:59.055824 26263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:59.055967 26262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:59.056617 26265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:59.057185 22946 server_base.cc:1034] running on GCE node
I20240502 20:46:59.057339 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:46:59.057374 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:46:59.057394 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682819057393 us; error 0 us; skew 500 ppm
I20240502 20:46:59.063548 22946 webserver.cc:460] Webserver started at http://127.22.104.190:45397/ using document root <none> and password file <none>
I20240502 20:46:59.063717 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:59.063769 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:59.063849 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:59.064258 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/master-0-root/instance:
uuid: "84ec0301760c45bf829f3eb53803135c"
format_stamp: "Formatted at 2024-05-02 20:46:59 on dist-test-slave-rgns"
I20240502 20:46:59.065651 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 20:46:59.066488 26270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:59.066764 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:59.066833 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/master-0-root
uuid: "84ec0301760c45bf829f3eb53803135c"
format_stamp: "Formatted at 2024-05-02 20:46:59 on dist-test-slave-rgns"
I20240502 20:46:59.066891 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:59.080722 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:59.081077 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:59.085139 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.190:39193
I20240502 20:46:59.087100 26322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.190:39193 every 8 connection(s)
I20240502 20:46:59.087500 26323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:59.089224 26323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84ec0301760c45bf829f3eb53803135c: Bootstrap starting.
I20240502 20:46:59.089917 26323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84ec0301760c45bf829f3eb53803135c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:59.090939 26323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84ec0301760c45bf829f3eb53803135c: No bootstrap required, opened a new log
I20240502 20:46:59.091284 26323 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84ec0301760c45bf829f3eb53803135c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ec0301760c45bf829f3eb53803135c" member_type: VOTER }
I20240502 20:46:59.091367 26323 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84ec0301760c45bf829f3eb53803135c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:59.091387 26323 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84ec0301760c45bf829f3eb53803135c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84ec0301760c45bf829f3eb53803135c, State: Initialized, Role: FOLLOWER
I20240502 20:46:59.091487 26323 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84ec0301760c45bf829f3eb53803135c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ec0301760c45bf829f3eb53803135c" member_type: VOTER }
I20240502 20:46:59.091539 26323 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84ec0301760c45bf829f3eb53803135c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:46:59.091564 26323 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84ec0301760c45bf829f3eb53803135c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:46:59.091595 26323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84ec0301760c45bf829f3eb53803135c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:59.092191 26323 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84ec0301760c45bf829f3eb53803135c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ec0301760c45bf829f3eb53803135c" member_type: VOTER }
I20240502 20:46:59.092298 26323 leader_election.cc:302] T 00000000000000000000000000000000 P 84ec0301760c45bf829f3eb53803135c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84ec0301760c45bf829f3eb53803135c; no voters:
I20240502 20:46:59.092432 26323 leader_election.cc:288] T 00000000000000000000000000000000 P 84ec0301760c45bf829f3eb53803135c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:46:59.092553 26326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84ec0301760c45bf829f3eb53803135c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:59.092747 26326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84ec0301760c45bf829f3eb53803135c [term 1 LEADER]: Becoming Leader. State: Replica: 84ec0301760c45bf829f3eb53803135c, State: Running, Role: LEADER
I20240502 20:46:59.092836 26323 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84ec0301760c45bf829f3eb53803135c [sys.catalog]: configured and running, proceeding with master startup.
I20240502 20:46:59.092867 26326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84ec0301760c45bf829f3eb53803135c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ec0301760c45bf829f3eb53803135c" member_type: VOTER }
I20240502 20:46:59.093272 26327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84ec0301760c45bf829f3eb53803135c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84ec0301760c45bf829f3eb53803135c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ec0301760c45bf829f3eb53803135c" member_type: VOTER } }
I20240502 20:46:59.093297 26328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84ec0301760c45bf829f3eb53803135c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84ec0301760c45bf829f3eb53803135c. Latest consensus state: current_term: 1 leader_uuid: "84ec0301760c45bf829f3eb53803135c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ec0301760c45bf829f3eb53803135c" member_type: VOTER } }
I20240502 20:46:59.093413 26328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84ec0301760c45bf829f3eb53803135c [sys.catalog]: This master's current role is: LEADER
I20240502 20:46:59.093634 26327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84ec0301760c45bf829f3eb53803135c [sys.catalog]: This master's current role is: LEADER
I20240502 20:46:59.094022 26332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 20:46:59.094962 26332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 20:46:59.095171 22946 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 20:46:59.096685 26332 catalog_manager.cc:1347] Generated new cluster ID: 047f7b93ddeb4009a841431bbe8c0a99
I20240502 20:46:59.096740 26332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 20:46:59.117712 26332 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 20:46:59.118250 26332 catalog_manager.cc:1504] Loading token signing keys...
I20240502 20:46:59.125049 26332 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 84ec0301760c45bf829f3eb53803135c: Generated new TSK 0
I20240502 20:46:59.125209 26332 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 20:46:59.127318 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:59.129202 26344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:59.129675 26345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:59.130180 26347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:59.130611 22946 server_base.cc:1034] running on GCE node
I20240502 20:46:59.130791 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:46:59.130831 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:46:59.130844 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682819130844 us; error 0 us; skew 500 ppm
I20240502 20:46:59.131711 22946 webserver.cc:460] Webserver started at http://127.22.104.129:40847/ using document root <none> and password file <none>
I20240502 20:46:59.131855 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:59.131899 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:59.131970 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:59.132351 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-0-root/instance:
uuid: "97406d5f0d1f4d59bdc784684baa1559"
format_stamp: "Formatted at 2024-05-02 20:46:59 on dist-test-slave-rgns"
I20240502 20:46:59.133744 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 20:46:59.134539 26352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:59.134802 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:59.134867 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-0-root
uuid: "97406d5f0d1f4d59bdc784684baa1559"
format_stamp: "Formatted at 2024-05-02 20:46:59 on dist-test-slave-rgns"
I20240502 20:46:59.134934 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:59.152304 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:59.152717 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:59.153004 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:59.153492 22946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:59.153532 22946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:59.153580 22946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:59.153613 22946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:59.157552 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.129:41989
I20240502 20:46:59.157589 26416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.129:41989 every 8 connection(s)
I20240502 20:46:59.158437 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:59.160216 26421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:59.160584 26424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:59.160856 26422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:59.161634 22946 server_base.cc:1034] running on GCE node
I20240502 20:46:59.161803 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:46:59.161842 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:46:59.161861 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682819161861 us; error 0 us; skew 500 ppm
I20240502 20:46:59.162813 22946 webserver.cc:460] Webserver started at http://127.22.104.130:33057/ using document root <none> and password file <none>
I20240502 20:46:59.162851 26417 heartbeater.cc:347] Connected to a master server at 127.22.104.190:39193
I20240502 20:46:59.162942 26417 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:59.162969 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:59.163010 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:59.163080 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:59.163177 26417 heartbeater.cc:511] Master 127.22.104.190:39193 requested a full tablet report, sending...
I20240502 20:46:59.163511 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-1-root/instance:
uuid: "ddb0f390a80f4c059c1a99a167c7d830"
format_stamp: "Formatted at 2024-05-02 20:46:59 on dist-test-slave-rgns"
I20240502 20:46:59.163832 26287 ts_manager.cc:194] Registered new tserver with Master: 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989)
I20240502 20:46:59.164531 26287 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58780
I20240502 20:46:59.165091 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 20:46:59.165958 26429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:59.166174 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:59.166235 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-1-root
uuid: "ddb0f390a80f4c059c1a99a167c7d830"
format_stamp: "Formatted at 2024-05-02 20:46:59 on dist-test-slave-rgns"
I20240502 20:46:59.166298 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:59.192456 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:59.192904 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:59.193253 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:59.193796 22946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:59.193840 22946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:59.193883 22946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:59.193949 22946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:59.198190 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.130:35519
I20240502 20:46:59.198251 26492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.130:35519 every 8 connection(s)
I20240502 20:46:59.198952 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:59.200804 26496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:59.200861 26497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:59.201602 26499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:59.202077 22946 server_base.cc:1034] running on GCE node
I20240502 20:46:59.202270 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:46:59.202320 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:46:59.202339 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682819202339 us; error 0 us; skew 500 ppm
I20240502 20:46:59.203109 26493 heartbeater.cc:347] Connected to a master server at 127.22.104.190:39193
I20240502 20:46:59.203212 26493 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:59.203439 26493 heartbeater.cc:511] Master 127.22.104.190:39193 requested a full tablet report, sending...
I20240502 20:46:59.203440 22946 webserver.cc:460] Webserver started at http://127.22.104.131:33987/ using document root <none> and password file <none>
I20240502 20:46:59.203634 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:59.203691 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:59.203765 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:59.204053 26287 ts_manager.cc:194] Registered new tserver with Master: ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519)
I20240502 20:46:59.204140 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-2-root/instance:
uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3"
format_stamp: "Formatted at 2024-05-02 20:46:59 on dist-test-slave-rgns"
I20240502 20:46:59.204589 26287 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58786
I20240502 20:46:59.205556 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 20:46:59.206425 26504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:59.206734 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:59.206799 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-2-root
uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3"
format_stamp: "Formatted at 2024-05-02 20:46:59 on dist-test-slave-rgns"
I20240502 20:46:59.206863 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:59.218544 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:59.218995 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:59.219276 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:59.219755 22946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:59.219792 22946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:59.219826 22946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:59.219854 22946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:59.224213 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.131:38881
I20240502 20:46:59.224267 26568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.131:38881 every 8 connection(s)
I20240502 20:46:59.228508 26569 heartbeater.cc:347] Connected to a master server at 127.22.104.190:39193
I20240502 20:46:59.228621 26569 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:59.228833 26569 heartbeater.cc:511] Master 127.22.104.190:39193 requested a full tablet report, sending...
I20240502 20:46:59.229432 26287 ts_manager.cc:194] Registered new tserver with Master: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881)
I20240502 20:46:59.229950 26287 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58794
I20240502 20:46:59.230223 22946 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005624788s
I20240502 20:47:00.165462 26417 heartbeater.cc:503] Master 127.22.104.190:39193 was elected leader, sending a full tablet report...
I20240502 20:47:00.205422 26493 heartbeater.cc:503] Master 127.22.104.190:39193 was elected leader, sending a full tablet report...
I20240502 20:47:00.230834 26569 heartbeater.cc:503] Master 127.22.104.190:39193 was elected leader, sending a full tablet report...
I20240502 20:47:00.256147 22946 test_util.cc:268] Using random seed: -1268293233
I20240502 20:47:00.264060 26287 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58798:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu""\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 20:47:00.264685 26287 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 20:47:00.278244 26529 tablet_service.cc:1467] Processing CreateTablet for tablet dfd1d70561924773aa204b5f22270b2e (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240502 20:47:00.278481 26533 tablet_service.cc:1467] Processing CreateTablet for tablet 32631c89419245dea46abe6baa5dc295 (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240502 20:47:00.278597 26529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfd1d70561924773aa204b5f22270b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.278926 26532 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7f28e11e1f42959251a7b03011f5dc (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240502 20:47:00.278926 26524 tablet_service.cc:1467] Processing CreateTablet for tablet 912fa9b5ccf74aadb519a8bf16659478 (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240502 20:47:00.279320 26531 tablet_service.cc:1467] Processing CreateTablet for tablet ca0df961aea849f4bc2ea340349f8848 (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240502 20:47:00.279687 26530 tablet_service.cc:1467] Processing CreateTablet for tablet 1409e44c98f44bd888eea4cee4f1163a (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240502 20:47:00.279845 26381 tablet_service.cc:1467] Processing CreateTablet for tablet 32631c89419245dea46abe6baa5dc295 (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240502 20:47:00.280143 26381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32631c89419245dea46abe6baa5dc295. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.280081 26377 tablet_service.cc:1467] Processing CreateTablet for tablet dfd1d70561924773aa204b5f22270b2e (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240502 20:47:00.280160 26526 tablet_service.cc:1467] Processing CreateTablet for tablet 1bc72d9e28894957a82f3b3ef37e6f08 (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240502 20:47:00.279845 26525 tablet_service.cc:1467] Processing CreateTablet for tablet 7d1ff41afa6e46609e6a3eb277df18ad (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240502 20:47:00.280371 26377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfd1d70561924773aa204b5f22270b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.280519 26527 tablet_service.cc:1467] Processing CreateTablet for tablet 082557bf92e4443cbb9d0c50a89d24c4 (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240502 20:47:00.280609 26379 tablet_service.cc:1467] Processing CreateTablet for tablet ca0df961aea849f4bc2ea340349f8848 (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240502 20:47:00.280825 26379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca0df961aea849f4bc2ea340349f8848. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.280858 26528 tablet_service.cc:1467] Processing CreateTablet for tablet 23f60af679e74f078fa50af755a9fbf6 (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240502 20:47:00.281152 26533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32631c89419245dea46abe6baa5dc295. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.280326 26380 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7f28e11e1f42959251a7b03011f5dc (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240502 20:47:00.282739 26380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7f28e11e1f42959251a7b03011f5dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.282838 26378 tablet_service.cc:1467] Processing CreateTablet for tablet 1409e44c98f44bd888eea4cee4f1163a (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240502 20:47:00.283039 26378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1409e44c98f44bd888eea4cee4f1163a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.283648 26528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23f60af679e74f078fa50af755a9fbf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.283792 26530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1409e44c98f44bd888eea4cee4f1163a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.284070 26526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bc72d9e28894957a82f3b3ef37e6f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.284365 26531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca0df961aea849f4bc2ea340349f8848. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.284638 26527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 082557bf92e4443cbb9d0c50a89d24c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.284860 26376 tablet_service.cc:1467] Processing CreateTablet for tablet 23f60af679e74f078fa50af755a9fbf6 (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240502 20:47:00.284968 26589 tablet_bootstrap.cc:492] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559: Bootstrap starting.
I20240502 20:47:00.285107 26376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23f60af679e74f078fa50af755a9fbf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.285241 26372 tablet_service.cc:1467] Processing CreateTablet for tablet 912fa9b5ccf74aadb519a8bf16659478 (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240502 20:47:00.285471 26372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 912fa9b5ccf74aadb519a8bf16659478. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.286101 26589 tablet_bootstrap.cc:654] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.286970 26375 tablet_service.cc:1467] Processing CreateTablet for tablet 082557bf92e4443cbb9d0c50a89d24c4 (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240502 20:47:00.287191 26589 tablet_bootstrap.cc:492] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559: No bootstrap required, opened a new log
I20240502 20:47:00.287218 26375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 082557bf92e4443cbb9d0c50a89d24c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.287274 26589 ts_tablet_manager.cc:1367] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:47:00.287535 26373 tablet_service.cc:1467] Processing CreateTablet for tablet 7d1ff41afa6e46609e6a3eb277df18ad (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240502 20:47:00.287654 26589 raft_consensus.cc:348] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } }
I20240502 20:47:00.287757 26373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d1ff41afa6e46609e6a3eb277df18ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.287760 26589 raft_consensus.cc:374] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.287825 26589 raft_consensus.cc:729] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97406d5f0d1f4d59bdc784684baa1559, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.287981 26589 consensus_queue.cc:260] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } }
I20240502 20:47:00.288336 26589 ts_tablet_manager.cc:1398] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:47:00.288554 26589 tablet_bootstrap.cc:492] T ca0df961aea849f4bc2ea340349f8848 P 97406d5f0d1f4d59bdc784684baa1559: Bootstrap starting.
I20240502 20:47:00.289398 26456 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7f28e11e1f42959251a7b03011f5dc (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240502 20:47:00.289634 26456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7f28e11e1f42959251a7b03011f5dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.289610 26374 tablet_service.cc:1467] Processing CreateTablet for tablet 1bc72d9e28894957a82f3b3ef37e6f08 (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240502 20:47:00.284937 26532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7f28e11e1f42959251a7b03011f5dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.289762 26455 tablet_service.cc:1467] Processing CreateTablet for tablet ca0df961aea849f4bc2ea340349f8848 (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240502 20:47:00.289844 26374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bc72d9e28894957a82f3b3ef37e6f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.289963 26455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca0df961aea849f4bc2ea340349f8848. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.291576 26524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 912fa9b5ccf74aadb519a8bf16659478. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.291941 26454 tablet_service.cc:1467] Processing CreateTablet for tablet 1409e44c98f44bd888eea4cee4f1163a (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240502 20:47:00.292167 26454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1409e44c98f44bd888eea4cee4f1163a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.293385 26525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d1ff41afa6e46609e6a3eb277df18ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.293995 26453 tablet_service.cc:1467] Processing CreateTablet for tablet dfd1d70561924773aa204b5f22270b2e (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240502 20:47:00.294241 26453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfd1d70561924773aa204b5f22270b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.294934 26452 tablet_service.cc:1467] Processing CreateTablet for tablet 23f60af679e74f078fa50af755a9fbf6 (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240502 20:47:00.295177 26452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23f60af679e74f078fa50af755a9fbf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.296432 26374 tablet_service.cc:1467] Processing CreateTablet for tablet 5db41d5853a34dc594bafb0f4c53cd7d (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240502 20:47:00.296659 26374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5db41d5853a34dc594bafb0f4c53cd7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.297034 26451 tablet_service.cc:1467] Processing CreateTablet for tablet 082557bf92e4443cbb9d0c50a89d24c4 (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240502 20:47:00.297259 26451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 082557bf92e4443cbb9d0c50a89d24c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.297657 26590 tablet_bootstrap.cc:492] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Bootstrap starting.
I20240502 20:47:00.298552 26590 tablet_bootstrap.cc:654] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.298900 26373 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea4b2412a3a420f93d384aa48117c60 (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240502 20:47:00.299103 26373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea4b2412a3a420f93d384aa48117c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.299124 26450 tablet_service.cc:1467] Processing CreateTablet for tablet 1bc72d9e28894957a82f3b3ef37e6f08 (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240502 20:47:00.299325 26450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bc72d9e28894957a82f3b3ef37e6f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.299599 26448 tablet_service.cc:1467] Processing CreateTablet for tablet 912fa9b5ccf74aadb519a8bf16659478 (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240502 20:47:00.299777 26448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 912fa9b5ccf74aadb519a8bf16659478. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.301201 26527 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea4b2412a3a420f93d384aa48117c60 (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240502 20:47:00.301201 26526 tablet_service.cc:1467] Processing CreateTablet for tablet 5db41d5853a34dc594bafb0f4c53cd7d (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240502 20:47:00.301280 26449 tablet_service.cc:1467] Processing CreateTablet for tablet 7d1ff41afa6e46609e6a3eb277df18ad (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240502 20:47:00.301452 26449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d1ff41afa6e46609e6a3eb277df18ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.301478 26527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea4b2412a3a420f93d384aa48117c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.301577 26526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5db41d5853a34dc594bafb0f4c53cd7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.289312 26457 tablet_service.cc:1467] Processing CreateTablet for tablet 32631c89419245dea46abe6baa5dc295 (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240502 20:47:00.303656 26457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32631c89419245dea46abe6baa5dc295. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.305334 26453 tablet_service.cc:1467] Processing CreateTablet for tablet 5db41d5853a34dc594bafb0f4c53cd7d (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240502 20:47:00.305564 26453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5db41d5853a34dc594bafb0f4c53cd7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.305624 26593 tablet_bootstrap.cc:492] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830: Bootstrap starting.
I20240502 20:47:00.305931 26589 tablet_bootstrap.cc:654] T ca0df961aea849f4bc2ea340349f8848 P 97406d5f0d1f4d59bdc784684baa1559: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.306463 26593 tablet_bootstrap.cc:654] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.305368 26449 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea4b2412a3a420f93d384aa48117c60 (DEFAULT_TABLE table=test-workload [id=cc2caf6496c0475da19150577095963d]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240502 20:47:00.307096 26590 tablet_bootstrap.cc:492] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: No bootstrap required, opened a new log
I20240502 20:47:00.307121 26449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea4b2412a3a420f93d384aa48117c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:00.307215 26590 ts_tablet_manager.cc:1367] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 20:47:00.307643 26593 tablet_bootstrap.cc:492] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830: No bootstrap required, opened a new log
I20240502 20:47:00.307724 26593 ts_tablet_manager.cc:1367] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:47:00.307951 26590 raft_consensus.cc:348] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } }
I20240502 20:47:00.308086 26593 raft_consensus.cc:348] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.308127 26590 raft_consensus.cc:374] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.308176 26593 raft_consensus.cc:374] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.308182 26590 raft_consensus.cc:729] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.308221 26593 raft_consensus.cc:729] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb0f390a80f4c059c1a99a167c7d830, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.308329 26593 consensus_queue.cc:260] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.307083 26589 tablet_bootstrap.cc:492] T ca0df961aea849f4bc2ea340349f8848 P 97406d5f0d1f4d59bdc784684baa1559: No bootstrap required, opened a new log
I20240502 20:47:00.308337 26590 consensus_queue.cc:260] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } }
I20240502 20:47:00.308501 26589 ts_tablet_manager.cc:1367] T ca0df961aea849f4bc2ea340349f8848 P 97406d5f0d1f4d59bdc784684baa1559: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240502 20:47:00.308684 26593 ts_tablet_manager.cc:1398] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:47:00.308765 26590 ts_tablet_manager.cc:1398] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:47:00.308851 26593 tablet_bootstrap.cc:492] T ca0df961aea849f4bc2ea340349f8848 P ddb0f390a80f4c059c1a99a167c7d830: Bootstrap starting.
I20240502 20:47:00.308879 26589 raft_consensus.cc:348] T ca0df961aea849f4bc2ea340349f8848 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.308957 26589 raft_consensus.cc:374] T ca0df961aea849f4bc2ea340349f8848 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.308984 26589 raft_consensus.cc:729] T ca0df961aea849f4bc2ea340349f8848 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97406d5f0d1f4d59bdc784684baa1559, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.309123 26589 consensus_queue.cc:260] T ca0df961aea849f4bc2ea340349f8848 P 97406d5f0d1f4d59bdc784684baa1559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.309351 26589 ts_tablet_manager.cc:1398] T ca0df961aea849f4bc2ea340349f8848 P 97406d5f0d1f4d59bdc784684baa1559: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:47:00.309729 26593 tablet_bootstrap.cc:654] T ca0df961aea849f4bc2ea340349f8848 P ddb0f390a80f4c059c1a99a167c7d830: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.309796 26589 tablet_bootstrap.cc:492] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559: Bootstrap starting.
I20240502 20:47:00.309888 26590 tablet_bootstrap.cc:492] T 7d7f28e11e1f42959251a7b03011f5dc P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Bootstrap starting.
I20240502 20:47:00.310614 26589 tablet_bootstrap.cc:654] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.310822 26590 tablet_bootstrap.cc:654] T 7d7f28e11e1f42959251a7b03011f5dc P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.310948 26593 tablet_bootstrap.cc:492] T ca0df961aea849f4bc2ea340349f8848 P ddb0f390a80f4c059c1a99a167c7d830: No bootstrap required, opened a new log
I20240502 20:47:00.311043 26593 ts_tablet_manager.cc:1367] T ca0df961aea849f4bc2ea340349f8848 P ddb0f390a80f4c059c1a99a167c7d830: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:47:00.311442 26593 raft_consensus.cc:348] T ca0df961aea849f4bc2ea340349f8848 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.311553 26593 raft_consensus.cc:374] T ca0df961aea849f4bc2ea340349f8848 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.311591 26593 raft_consensus.cc:729] T ca0df961aea849f4bc2ea340349f8848 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb0f390a80f4c059c1a99a167c7d830, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.311697 26593 consensus_queue.cc:260] T ca0df961aea849f4bc2ea340349f8848 P ddb0f390a80f4c059c1a99a167c7d830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.311926 26593 ts_tablet_manager.cc:1398] T ca0df961aea849f4bc2ea340349f8848 P ddb0f390a80f4c059c1a99a167c7d830: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:47:00.311959 26589 tablet_bootstrap.cc:492] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559: No bootstrap required, opened a new log
I20240502 20:47:00.312031 26589 ts_tablet_manager.cc:1367] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 20:47:00.312065 26593 tablet_bootstrap.cc:492] T 1409e44c98f44bd888eea4cee4f1163a P ddb0f390a80f4c059c1a99a167c7d830: Bootstrap starting.
I20240502 20:47:00.312419 26589 raft_consensus.cc:348] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.312506 26589 raft_consensus.cc:374] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.312538 26589 raft_consensus.cc:729] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97406d5f0d1f4d59bdc784684baa1559, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.312619 26590 tablet_bootstrap.cc:492] T 7d7f28e11e1f42959251a7b03011f5dc P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: No bootstrap required, opened a new log
I20240502 20:47:00.312687 26590 ts_tablet_manager.cc:1367] T 7d7f28e11e1f42959251a7b03011f5dc P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 20:47:00.312647 26589 consensus_queue.cc:260] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.312873 26593 tablet_bootstrap.cc:654] T 1409e44c98f44bd888eea4cee4f1163a P ddb0f390a80f4c059c1a99a167c7d830: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.312882 26589 ts_tablet_manager.cc:1398] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:47:00.313107 26590 raft_consensus.cc:348] T 7d7f28e11e1f42959251a7b03011f5dc P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.313223 26590 raft_consensus.cc:374] T 7d7f28e11e1f42959251a7b03011f5dc P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.313252 26590 raft_consensus.cc:729] T 7d7f28e11e1f42959251a7b03011f5dc P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.313356 26590 consensus_queue.cc:260] T 7d7f28e11e1f42959251a7b03011f5dc P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.313608 26590 ts_tablet_manager.cc:1398] T 7d7f28e11e1f42959251a7b03011f5dc P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:47:00.313746 26590 tablet_bootstrap.cc:492] T dfd1d70561924773aa204b5f22270b2e P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Bootstrap starting.
I20240502 20:47:00.313104 26589 tablet_bootstrap.cc:492] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559: Bootstrap starting.
I20240502 20:47:00.314561 26589 tablet_bootstrap.cc:654] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.314561 26590 tablet_bootstrap.cc:654] T dfd1d70561924773aa204b5f22270b2e P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.315639 26590 tablet_bootstrap.cc:492] T dfd1d70561924773aa204b5f22270b2e P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: No bootstrap required, opened a new log
I20240502 20:47:00.315721 26590 ts_tablet_manager.cc:1367] T dfd1d70561924773aa204b5f22270b2e P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 20:47:00.315933 26589 tablet_bootstrap.cc:492] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559: No bootstrap required, opened a new log
I20240502 20:47:00.316004 26589 ts_tablet_manager.cc:1367] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 20:47:00.316078 26590 raft_consensus.cc:348] T dfd1d70561924773aa204b5f22270b2e P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.316146 26590 raft_consensus.cc:374] T dfd1d70561924773aa204b5f22270b2e P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.316164 26590 raft_consensus.cc:729] T dfd1d70561924773aa204b5f22270b2e P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.316255 26590 consensus_queue.cc:260] T dfd1d70561924773aa204b5f22270b2e P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.316370 26589 raft_consensus.cc:348] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.316458 26589 raft_consensus.cc:374] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.316478 26590 ts_tablet_manager.cc:1398] T dfd1d70561924773aa204b5f22270b2e P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:47:00.316620 26590 tablet_bootstrap.cc:492] T 912fa9b5ccf74aadb519a8bf16659478 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Bootstrap starting.
I20240502 20:47:00.317337 26590 tablet_bootstrap.cc:654] T 912fa9b5ccf74aadb519a8bf16659478 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.316488 26589 raft_consensus.cc:729] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97406d5f0d1f4d59bdc784684baa1559, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.318011 26589 consensus_queue.cc:260] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.318264 26589 ts_tablet_manager.cc:1398] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:47:00.318291 26590 tablet_bootstrap.cc:492] T 912fa9b5ccf74aadb519a8bf16659478 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: No bootstrap required, opened a new log
I20240502 20:47:00.318362 26590 ts_tablet_manager.cc:1367] T 912fa9b5ccf74aadb519a8bf16659478 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:47:00.318424 26589 tablet_bootstrap.cc:492] T 1409e44c98f44bd888eea4cee4f1163a P 97406d5f0d1f4d59bdc784684baa1559: Bootstrap starting.
I20240502 20:47:00.318686 26590 raft_consensus.cc:348] T 912fa9b5ccf74aadb519a8bf16659478 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.318842 26590 raft_consensus.cc:374] T 912fa9b5ccf74aadb519a8bf16659478 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.318899 26590 raft_consensus.cc:729] T 912fa9b5ccf74aadb519a8bf16659478 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.319058 26590 consensus_queue.cc:260] T 912fa9b5ccf74aadb519a8bf16659478 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.319200 26593 tablet_bootstrap.cc:492] T 1409e44c98f44bd888eea4cee4f1163a P ddb0f390a80f4c059c1a99a167c7d830: No bootstrap required, opened a new log
I20240502 20:47:00.319239 26589 tablet_bootstrap.cc:654] T 1409e44c98f44bd888eea4cee4f1163a P 97406d5f0d1f4d59bdc784684baa1559: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.319294 26593 ts_tablet_manager.cc:1367] T 1409e44c98f44bd888eea4cee4f1163a P ddb0f390a80f4c059c1a99a167c7d830: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240502 20:47:00.319576 26590 ts_tablet_manager.cc:1398] T 912fa9b5ccf74aadb519a8bf16659478 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:47:00.320156 26589 tablet_bootstrap.cc:492] T 1409e44c98f44bd888eea4cee4f1163a P 97406d5f0d1f4d59bdc784684baa1559: No bootstrap required, opened a new log
I20240502 20:47:00.320232 26589 ts_tablet_manager.cc:1367] T 1409e44c98f44bd888eea4cee4f1163a P 97406d5f0d1f4d59bdc784684baa1559: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 20:47:00.320297 26590 tablet_bootstrap.cc:492] T 23f60af679e74f078fa50af755a9fbf6 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Bootstrap starting.
I20240502 20:47:00.320621 26589 raft_consensus.cc:348] T 1409e44c98f44bd888eea4cee4f1163a P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.320708 26589 raft_consensus.cc:374] T 1409e44c98f44bd888eea4cee4f1163a P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.320740 26589 raft_consensus.cc:729] T 1409e44c98f44bd888eea4cee4f1163a P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97406d5f0d1f4d59bdc784684baa1559, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.319691 26593 raft_consensus.cc:348] T 1409e44c98f44bd888eea4cee4f1163a P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.320839 26589 consensus_queue.cc:260] T 1409e44c98f44bd888eea4cee4f1163a P 97406d5f0d1f4d59bdc784684baa1559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.320909 26593 raft_consensus.cc:374] T 1409e44c98f44bd888eea4cee4f1163a P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.320945 26593 raft_consensus.cc:729] T 1409e44c98f44bd888eea4cee4f1163a P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb0f390a80f4c059c1a99a167c7d830, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.321005 26590 tablet_bootstrap.cc:654] T 23f60af679e74f078fa50af755a9fbf6 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.321069 26589 ts_tablet_manager.cc:1398] T 1409e44c98f44bd888eea4cee4f1163a P 97406d5f0d1f4d59bdc784684baa1559: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:47:00.321149 26593 consensus_queue.cc:260] T 1409e44c98f44bd888eea4cee4f1163a P ddb0f390a80f4c059c1a99a167c7d830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.321216 26589 tablet_bootstrap.cc:492] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559: Bootstrap starting.
I20240502 20:47:00.321383 26593 ts_tablet_manager.cc:1398] T 1409e44c98f44bd888eea4cee4f1163a P ddb0f390a80f4c059c1a99a167c7d830: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:47:00.321806 26593 tablet_bootstrap.cc:492] T 23f60af679e74f078fa50af755a9fbf6 P ddb0f390a80f4c059c1a99a167c7d830: Bootstrap starting.
I20240502 20:47:00.322019 26589 tablet_bootstrap.cc:654] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.322618 26593 tablet_bootstrap.cc:654] T 23f60af679e74f078fa50af755a9fbf6 P ddb0f390a80f4c059c1a99a167c7d830: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.322997 26589 tablet_bootstrap.cc:492] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559: No bootstrap required, opened a new log
I20240502 20:47:00.323081 26589 ts_tablet_manager.cc:1367] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:47:00.323453 26589 raft_consensus.cc:348] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.323546 26589 raft_consensus.cc:374] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.323578 26589 raft_consensus.cc:729] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97406d5f0d1f4d59bdc784684baa1559, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.323654 26593 tablet_bootstrap.cc:492] T 23f60af679e74f078fa50af755a9fbf6 P ddb0f390a80f4c059c1a99a167c7d830: No bootstrap required, opened a new log
I20240502 20:47:00.323734 26593 ts_tablet_manager.cc:1367] T 23f60af679e74f078fa50af755a9fbf6 P ddb0f390a80f4c059c1a99a167c7d830: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:47:00.324138 26593 raft_consensus.cc:348] T 23f60af679e74f078fa50af755a9fbf6 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.324196 26590 tablet_bootstrap.cc:492] T 23f60af679e74f078fa50af755a9fbf6 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: No bootstrap required, opened a new log
I20240502 20:47:00.324244 26593 raft_consensus.cc:374] T 23f60af679e74f078fa50af755a9fbf6 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.324281 26593 raft_consensus.cc:729] T 23f60af679e74f078fa50af755a9fbf6 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb0f390a80f4c059c1a99a167c7d830, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.324282 26590 ts_tablet_manager.cc:1367] T 23f60af679e74f078fa50af755a9fbf6 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:47:00.324390 26593 consensus_queue.cc:260] T 23f60af679e74f078fa50af755a9fbf6 P ddb0f390a80f4c059c1a99a167c7d830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.324620 26593 ts_tablet_manager.cc:1398] T 23f60af679e74f078fa50af755a9fbf6 P ddb0f390a80f4c059c1a99a167c7d830: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:47:00.324648 26590 raft_consensus.cc:348] T 23f60af679e74f078fa50af755a9fbf6 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.324769 26590 raft_consensus.cc:374] T 23f60af679e74f078fa50af755a9fbf6 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.324771 26593 tablet_bootstrap.cc:492] T 082557bf92e4443cbb9d0c50a89d24c4 P ddb0f390a80f4c059c1a99a167c7d830: Bootstrap starting.
I20240502 20:47:00.324853 26590 raft_consensus.cc:729] T 23f60af679e74f078fa50af755a9fbf6 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.323693 26589 consensus_queue.cc:260] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.324985 26590 consensus_queue.cc:260] T 23f60af679e74f078fa50af755a9fbf6 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.326270 26590 ts_tablet_manager.cc:1398] T 23f60af679e74f078fa50af755a9fbf6 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:47:00.325284 26589 ts_tablet_manager.cc:1398] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:47:00.325680 26593 tablet_bootstrap.cc:654] T 082557bf92e4443cbb9d0c50a89d24c4 P ddb0f390a80f4c059c1a99a167c7d830: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.326473 26589 tablet_bootstrap.cc:492] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559: Bootstrap starting.
I20240502 20:47:00.327311 26589 tablet_bootstrap.cc:654] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.327422 26593 tablet_bootstrap.cc:492] T 082557bf92e4443cbb9d0c50a89d24c4 P ddb0f390a80f4c059c1a99a167c7d830: No bootstrap required, opened a new log
I20240502 20:47:00.327517 26593 ts_tablet_manager.cc:1367] T 082557bf92e4443cbb9d0c50a89d24c4 P ddb0f390a80f4c059c1a99a167c7d830: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:47:00.327893 26593 raft_consensus.cc:348] T 082557bf92e4443cbb9d0c50a89d24c4 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.327994 26593 raft_consensus.cc:374] T 082557bf92e4443cbb9d0c50a89d24c4 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.328039 26590 tablet_bootstrap.cc:492] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Bootstrap starting.
I20240502 20:47:00.328051 26593 raft_consensus.cc:729] T 082557bf92e4443cbb9d0c50a89d24c4 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb0f390a80f4c059c1a99a167c7d830, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.328276 26593 consensus_queue.cc:260] T 082557bf92e4443cbb9d0c50a89d24c4 P ddb0f390a80f4c059c1a99a167c7d830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.328580 26589 tablet_bootstrap.cc:492] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559: No bootstrap required, opened a new log
I20240502 20:47:00.328661 26589 ts_tablet_manager.cc:1367] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:47:00.328936 26590 tablet_bootstrap.cc:654] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.329061 26593 ts_tablet_manager.cc:1398] T 082557bf92e4443cbb9d0c50a89d24c4 P ddb0f390a80f4c059c1a99a167c7d830: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:47:00.329051 26589 raft_consensus.cc:348] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.329180 26589 raft_consensus.cc:374] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.329211 26589 raft_consensus.cc:729] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97406d5f0d1f4d59bdc784684baa1559, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.329263 26593 tablet_bootstrap.cc:492] T 1bc72d9e28894957a82f3b3ef37e6f08 P ddb0f390a80f4c059c1a99a167c7d830: Bootstrap starting.
I20240502 20:47:00.329315 26589 consensus_queue.cc:260] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.329542 26589 ts_tablet_manager.cc:1398] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:47:00.329699 26589 tablet_bootstrap.cc:492] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559: Bootstrap starting.
I20240502 20:47:00.330008 26593 tablet_bootstrap.cc:654] T 1bc72d9e28894957a82f3b3ef37e6f08 P ddb0f390a80f4c059c1a99a167c7d830: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.330525 26589 tablet_bootstrap.cc:654] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.330909 26593 tablet_bootstrap.cc:492] T 1bc72d9e28894957a82f3b3ef37e6f08 P ddb0f390a80f4c059c1a99a167c7d830: No bootstrap required, opened a new log
I20240502 20:47:00.330968 26593 ts_tablet_manager.cc:1367] T 1bc72d9e28894957a82f3b3ef37e6f08 P ddb0f390a80f4c059c1a99a167c7d830: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:47:00.331283 26593 raft_consensus.cc:348] T 1bc72d9e28894957a82f3b3ef37e6f08 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.331374 26593 raft_consensus.cc:374] T 1bc72d9e28894957a82f3b3ef37e6f08 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.331430 26593 raft_consensus.cc:729] T 1bc72d9e28894957a82f3b3ef37e6f08 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb0f390a80f4c059c1a99a167c7d830, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.331576 26589 tablet_bootstrap.cc:492] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559: No bootstrap required, opened a new log
I20240502 20:47:00.331562 26593 consensus_queue.cc:260] T 1bc72d9e28894957a82f3b3ef37e6f08 P ddb0f390a80f4c059c1a99a167c7d830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.331676 26589 ts_tablet_manager.cc:1367] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:47:00.332098 26593 ts_tablet_manager.cc:1398] T 1bc72d9e28894957a82f3b3ef37e6f08 P ddb0f390a80f4c059c1a99a167c7d830: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:47:00.332069 26589 raft_consensus.cc:348] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.332250 26589 raft_consensus.cc:374] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.332290 26589 raft_consensus.cc:729] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97406d5f0d1f4d59bdc784684baa1559, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.332336 26593 tablet_bootstrap.cc:492] T 912fa9b5ccf74aadb519a8bf16659478 P ddb0f390a80f4c059c1a99a167c7d830: Bootstrap starting.
I20240502 20:47:00.332382 26589 consensus_queue.cc:260] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.332628 26589 ts_tablet_manager.cc:1398] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:47:00.332782 26589 tablet_bootstrap.cc:492] T 7d1ff41afa6e46609e6a3eb277df18ad P 97406d5f0d1f4d59bdc784684baa1559: Bootstrap starting.
I20240502 20:47:00.333210 26593 tablet_bootstrap.cc:654] T 912fa9b5ccf74aadb519a8bf16659478 P ddb0f390a80f4c059c1a99a167c7d830: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.333633 26589 tablet_bootstrap.cc:654] T 7d1ff41afa6e46609e6a3eb277df18ad P 97406d5f0d1f4d59bdc784684baa1559: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.334185 26592 raft_consensus.cc:482] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:47:00.334276 26592 raft_consensus.cc:504] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.334762 26592 leader_election.cc:288] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519), bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881)
I20240502 20:47:00.334803 26590 tablet_bootstrap.cc:492] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: No bootstrap required, opened a new log
I20240502 20:47:00.334872 26590 ts_tablet_manager.cc:1367] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240502 20:47:00.334918 26589 tablet_bootstrap.cc:492] T 7d1ff41afa6e46609e6a3eb277df18ad P 97406d5f0d1f4d59bdc784684baa1559: No bootstrap required, opened a new log
I20240502 20:47:00.334975 26589 ts_tablet_manager.cc:1367] T 7d1ff41afa6e46609e6a3eb277df18ad P 97406d5f0d1f4d59bdc784684baa1559: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 20:47:00.335261 26590 raft_consensus.cc:348] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.335350 26590 raft_consensus.cc:374] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.335381 26590 raft_consensus.cc:729] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.335326 26589 raft_consensus.cc:348] T 7d1ff41afa6e46609e6a3eb277df18ad P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.335427 26589 raft_consensus.cc:374] T 7d1ff41afa6e46609e6a3eb277df18ad P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.335460 26589 raft_consensus.cc:729] T 7d1ff41afa6e46609e6a3eb277df18ad P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97406d5f0d1f4d59bdc784684baa1559, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.335491 26590 consensus_queue.cc:260] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.335552 26589 consensus_queue.cc:260] T 7d1ff41afa6e46609e6a3eb277df18ad P 97406d5f0d1f4d59bdc784684baa1559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.335747 26590 ts_tablet_manager.cc:1398] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:47:00.335762 26589 ts_tablet_manager.cc:1398] T 7d1ff41afa6e46609e6a3eb277df18ad P 97406d5f0d1f4d59bdc784684baa1559: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:47:00.336028 26590 tablet_bootstrap.cc:492] T 7d1ff41afa6e46609e6a3eb277df18ad P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Bootstrap starting.
I20240502 20:47:00.336067 26589 tablet_bootstrap.cc:492] T 1bc72d9e28894957a82f3b3ef37e6f08 P 97406d5f0d1f4d59bdc784684baa1559: Bootstrap starting.
I20240502 20:47:00.336833 26589 tablet_bootstrap.cc:654] T 1bc72d9e28894957a82f3b3ef37e6f08 P 97406d5f0d1f4d59bdc784684baa1559: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.336833 26590 tablet_bootstrap.cc:654] T 7d1ff41afa6e46609e6a3eb277df18ad P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.337814 26589 tablet_bootstrap.cc:492] T 1bc72d9e28894957a82f3b3ef37e6f08 P 97406d5f0d1f4d59bdc784684baa1559: No bootstrap required, opened a new log
I20240502 20:47:00.337886 26589 ts_tablet_manager.cc:1367] T 1bc72d9e28894957a82f3b3ef37e6f08 P 97406d5f0d1f4d59bdc784684baa1559: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:47:00.338284 26589 raft_consensus.cc:348] T 1bc72d9e28894957a82f3b3ef37e6f08 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.338378 26589 raft_consensus.cc:374] T 1bc72d9e28894957a82f3b3ef37e6f08 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.338410 26589 raft_consensus.cc:729] T 1bc72d9e28894957a82f3b3ef37e6f08 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97406d5f0d1f4d59bdc784684baa1559, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.338513 26589 consensus_queue.cc:260] T 1bc72d9e28894957a82f3b3ef37e6f08 P 97406d5f0d1f4d59bdc784684baa1559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.338897 26589 ts_tablet_manager.cc:1398] T 1bc72d9e28894957a82f3b3ef37e6f08 P 97406d5f0d1f4d59bdc784684baa1559: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:47:00.339097 26589 tablet_bootstrap.cc:492] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559: Bootstrap starting.
I20240502 20:47:00.339953 26589 tablet_bootstrap.cc:654] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.340309 26467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7f28e11e1f42959251a7b03011f5dc" candidate_uuid: "97406d5f0d1f4d59bdc784684baa1559" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830" is_pre_election: true
I20240502 20:47:00.340425 26467 raft_consensus.cc:2454] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97406d5f0d1f4d59bdc784684baa1559 in term 0.
I20240502 20:47:00.340576 26590 tablet_bootstrap.cc:492] T 7d1ff41afa6e46609e6a3eb277df18ad P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: No bootstrap required, opened a new log
I20240502 20:47:00.340662 26590 ts_tablet_manager.cc:1367] T 7d1ff41afa6e46609e6a3eb277df18ad P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 20:47:00.340744 26353 leader_election.cc:302] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97406d5f0d1f4d59bdc784684baa1559, ddb0f390a80f4c059c1a99a167c7d830; no voters:
I20240502 20:47:00.340915 26592 raft_consensus.cc:2781] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:47:00.340972 26592 raft_consensus.cc:482] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:47:00.341008 26592 raft_consensus.cc:3037] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.341043 26590 raft_consensus.cc:348] T 7d1ff41afa6e46609e6a3eb277df18ad P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.341121 26590 raft_consensus.cc:374] T 7d1ff41afa6e46609e6a3eb277df18ad P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.341147 26590 raft_consensus.cc:729] T 7d1ff41afa6e46609e6a3eb277df18ad P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.341253 26590 consensus_queue.cc:260] T 7d1ff41afa6e46609e6a3eb277df18ad P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.341476 26589 tablet_bootstrap.cc:492] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559: No bootstrap required, opened a new log
I20240502 20:47:00.341528 26590 ts_tablet_manager.cc:1398] T 7d1ff41afa6e46609e6a3eb277df18ad P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:47:00.341552 26589 ts_tablet_manager.cc:1367] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:47:00.341704 26590 tablet_bootstrap.cc:492] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Bootstrap starting.
I20240502 20:47:00.341933 26589 raft_consensus.cc:348] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.342033 26589 raft_consensus.cc:374] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.342021 26592 raft_consensus.cc:504] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.342096 26589 raft_consensus.cc:729] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97406d5f0d1f4d59bdc784684baa1559, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.342240 26589 consensus_queue.cc:260] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.342391 26592 leader_election.cc:288] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [CANDIDATE]: Term 1 election: Requested vote from peers ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519), bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881)
I20240502 20:47:00.342538 26590 tablet_bootstrap.cc:654] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.342584 26589 ts_tablet_manager.cc:1398] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 20:47:00.342761 26589 tablet_bootstrap.cc:492] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559: Bootstrap starting.
I20240502 20:47:00.343541 26467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7f28e11e1f42959251a7b03011f5dc" candidate_uuid: "97406d5f0d1f4d59bdc784684baa1559" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830"
I20240502 20:47:00.343634 26589 tablet_bootstrap.cc:654] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.343652 26467 raft_consensus.cc:3037] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.344808 26467 raft_consensus.cc:2454] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97406d5f0d1f4d59bdc784684baa1559 in term 1.
I20240502 20:47:00.345130 26543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7f28e11e1f42959251a7b03011f5dc" candidate_uuid: "97406d5f0d1f4d59bdc784684baa1559" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" is_pre_election: true
I20240502 20:47:00.345253 26543 raft_consensus.cc:2454] T 7d7f28e11e1f42959251a7b03011f5dc P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97406d5f0d1f4d59bdc784684baa1559 in term 0.
I20240502 20:47:00.345155 26353 leader_election.cc:302] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97406d5f0d1f4d59bdc784684baa1559, ddb0f390a80f4c059c1a99a167c7d830; no voters:
I20240502 20:47:00.345484 26542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7f28e11e1f42959251a7b03011f5dc" candidate_uuid: "97406d5f0d1f4d59bdc784684baa1559" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3"
I20240502 20:47:00.345546 26590 tablet_bootstrap.cc:492] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: No bootstrap required, opened a new log
I20240502 20:47:00.345594 26542 raft_consensus.cc:3037] T 7d7f28e11e1f42959251a7b03011f5dc P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.345623 26590 ts_tablet_manager.cc:1367] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 20:47:00.346061 26590 raft_consensus.cc:348] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.346163 26590 raft_consensus.cc:374] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.346192 26590 raft_consensus.cc:729] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.346236 26622 raft_consensus.cc:2781] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:47:00.346328 26542 raft_consensus.cc:2454] T 7d7f28e11e1f42959251a7b03011f5dc P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97406d5f0d1f4d59bdc784684baa1559 in term 1.
I20240502 20:47:00.346307 26590 consensus_queue.cc:260] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.346406 26622 raft_consensus.cc:686] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [term 1 LEADER]: Becoming Leader. State: Replica: 97406d5f0d1f4d59bdc784684baa1559, State: Running, Role: LEADER
I20240502 20:47:00.346566 26590 ts_tablet_manager.cc:1398] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:47:00.346529 26622 consensus_queue.cc:237] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.346716 26593 tablet_bootstrap.cc:492] T 912fa9b5ccf74aadb519a8bf16659478 P ddb0f390a80f4c059c1a99a167c7d830: No bootstrap required, opened a new log
I20240502 20:47:00.346737 26590 tablet_bootstrap.cc:492] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Bootstrap starting.
I20240502 20:47:00.346107 26592 raft_consensus.cc:482] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:47:00.346791 26593 ts_tablet_manager.cc:1367] T 912fa9b5ccf74aadb519a8bf16659478 P ddb0f390a80f4c059c1a99a167c7d830: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240502 20:47:00.346840 26592 raft_consensus.cc:504] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.347172 26592 leader_election.cc:288] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519), bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881)
I20240502 20:47:00.347152 26593 raft_consensus.cc:348] T 912fa9b5ccf74aadb519a8bf16659478 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.347237 26593 raft_consensus.cc:374] T 912fa9b5ccf74aadb519a8bf16659478 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.347265 26593 raft_consensus.cc:729] T 912fa9b5ccf74aadb519a8bf16659478 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb0f390a80f4c059c1a99a167c7d830, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.347362 26593 consensus_queue.cc:260] T 912fa9b5ccf74aadb519a8bf16659478 P ddb0f390a80f4c059c1a99a167c7d830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.347385 26467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5db41d5853a34dc594bafb0f4c53cd7d" candidate_uuid: "97406d5f0d1f4d59bdc784684baa1559" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830" is_pre_election: true
I20240502 20:47:00.347589 26590 tablet_bootstrap.cc:654] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.347718 26542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5db41d5853a34dc594bafb0f4c53cd7d" candidate_uuid: "97406d5f0d1f4d59bdc784684baa1559" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" is_pre_election: true
W20240502 20:47:00.348012 26354 leader_election.cc:341] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881): Illegal state: must be running to vote when last-logged opid is not known
I20240502 20:47:00.348215 26593 ts_tablet_manager.cc:1398] T 912fa9b5ccf74aadb519a8bf16659478 P ddb0f390a80f4c059c1a99a167c7d830: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:47:00.348601 26590 tablet_bootstrap.cc:492] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: No bootstrap required, opened a new log
I20240502 20:47:00.348685 26590 ts_tablet_manager.cc:1367] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 20:47:00.348868 26589 tablet_bootstrap.cc:492] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559: No bootstrap required, opened a new log
I20240502 20:47:00.348939 26589 ts_tablet_manager.cc:1367] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
W20240502 20:47:00.348994 26353 leader_election.cc:341] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519): Illegal state: must be running to vote when last-logged opid is not known
I20240502 20:47:00.349056 26353 leader_election.cc:302] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97406d5f0d1f4d59bdc784684baa1559; no voters: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, ddb0f390a80f4c059c1a99a167c7d830
I20240502 20:47:00.349078 26590 raft_consensus.cc:348] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.349154 26590 raft_consensus.cc:374] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.349182 26590 raft_consensus.cc:729] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.349280 26590 consensus_queue.cc:260] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.349337 26589 raft_consensus.cc:348] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.349409 26593 tablet_bootstrap.cc:492] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830: Bootstrap starting.
I20240502 20:47:00.349431 26589 raft_consensus.cc:374] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.349460 26589 raft_consensus.cc:729] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97406d5f0d1f4d59bdc784684baa1559, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.349529 26590 ts_tablet_manager.cc:1398] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:47:00.349550 26592 raft_consensus.cc:2726] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 20:47:00.349565 26589 consensus_queue.cc:260] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.349689 26590 tablet_bootstrap.cc:492] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Bootstrap starting.
I20240502 20:47:00.349797 26589 ts_tablet_manager.cc:1398] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:47:00.350219 26593 tablet_bootstrap.cc:654] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.350539 26590 tablet_bootstrap.cc:654] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.351859 26287 catalog_manager.cc:5484] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129). New cstate: current_term: 1 leader_uuid: "97406d5f0d1f4d59bdc784684baa1559" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:00.352111 26590 tablet_bootstrap.cc:492] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: No bootstrap required, opened a new log
I20240502 20:47:00.352200 26590 ts_tablet_manager.cc:1367] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 20:47:00.352586 26590 raft_consensus.cc:348] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.352687 26590 raft_consensus.cc:374] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.352715 26590 raft_consensus.cc:729] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.352820 26590 consensus_queue.cc:260] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.353067 26590 ts_tablet_manager.cc:1398] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:47:00.353230 26590 tablet_bootstrap.cc:492] T 5db41d5853a34dc594bafb0f4c53cd7d P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Bootstrap starting.
I20240502 20:47:00.354084 26590 tablet_bootstrap.cc:654] T 5db41d5853a34dc594bafb0f4c53cd7d P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.355108 26590 tablet_bootstrap.cc:492] T 5db41d5853a34dc594bafb0f4c53cd7d P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: No bootstrap required, opened a new log
I20240502 20:47:00.355188 26590 ts_tablet_manager.cc:1367] T 5db41d5853a34dc594bafb0f4c53cd7d P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 20:47:00.355540 26590 raft_consensus.cc:348] T 5db41d5853a34dc594bafb0f4c53cd7d P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.355609 26590 raft_consensus.cc:374] T 5db41d5853a34dc594bafb0f4c53cd7d P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.355626 26590 raft_consensus.cc:729] T 5db41d5853a34dc594bafb0f4c53cd7d P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.355706 26590 consensus_queue.cc:260] T 5db41d5853a34dc594bafb0f4c53cd7d P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.355974 26590 ts_tablet_manager.cc:1398] T 5db41d5853a34dc594bafb0f4c53cd7d P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:47:00.356247 26590 tablet_bootstrap.cc:492] T 8ea4b2412a3a420f93d384aa48117c60 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Bootstrap starting.
I20240502 20:47:00.356967 26598 raft_consensus.cc:482] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:47:00.357017 26590 tablet_bootstrap.cc:654] T 8ea4b2412a3a420f93d384aa48117c60 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.357054 26598 raft_consensus.cc:504] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.357602 26598 leader_election.cc:288] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989), ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519)
I20240502 20:47:00.358237 26593 tablet_bootstrap.cc:492] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830: No bootstrap required, opened a new log
I20240502 20:47:00.358371 26593 ts_tablet_manager.cc:1367] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 20:47:00.358582 26590 tablet_bootstrap.cc:492] T 8ea4b2412a3a420f93d384aa48117c60 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: No bootstrap required, opened a new log
I20240502 20:47:00.358667 26590 ts_tablet_manager.cc:1367] T 8ea4b2412a3a420f93d384aa48117c60 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:47:00.358908 26593 raft_consensus.cc:348] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.359082 26593 raft_consensus.cc:374] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.359117 26590 raft_consensus.cc:348] T 8ea4b2412a3a420f93d384aa48117c60 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.359165 26593 raft_consensus.cc:729] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb0f390a80f4c059c1a99a167c7d830, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.359263 26590 raft_consensus.cc:374] T 8ea4b2412a3a420f93d384aa48117c60 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.359367 26590 raft_consensus.cc:729] T 8ea4b2412a3a420f93d384aa48117c60 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.359483 26593 consensus_queue.cc:260] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.359530 26590 consensus_queue.cc:260] T 8ea4b2412a3a420f93d384aa48117c60 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.359936 26590 ts_tablet_manager.cc:1398] T 8ea4b2412a3a420f93d384aa48117c60 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:47:00.360334 26593 ts_tablet_manager.cc:1398] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:47:00.360524 26593 tablet_bootstrap.cc:492] T dfd1d70561924773aa204b5f22270b2e P ddb0f390a80f4c059c1a99a167c7d830: Bootstrap starting.
I20240502 20:47:00.362459 26391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1409e44c98f44bd888eea4cee4f1163a" candidate_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97406d5f0d1f4d59bdc784684baa1559" is_pre_election: true
I20240502 20:47:00.362589 26391 raft_consensus.cc:2454] T 1409e44c98f44bd888eea4cee4f1163a P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 in term 0.
I20240502 20:47:00.362792 26593 tablet_bootstrap.cc:654] T dfd1d70561924773aa204b5f22270b2e P ddb0f390a80f4c059c1a99a167c7d830: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.362915 26507 leader_election.cc:302] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97406d5f0d1f4d59bdc784684baa1559, bdcd7bbd5fa14f21bbf0a7a3cdc33cb3; no voters:
I20240502 20:47:00.363106 26598 raft_consensus.cc:2781] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:47:00.363168 26598 raft_consensus.cc:482] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:47:00.363138 26467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1409e44c98f44bd888eea4cee4f1163a" candidate_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830" is_pre_election: true
I20240502 20:47:00.363225 26598 raft_consensus.cc:3037] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.363267 26467 raft_consensus.cc:2454] T 1409e44c98f44bd888eea4cee4f1163a P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 in term 0.
I20240502 20:47:00.364204 26598 raft_consensus.cc:504] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.364365 26593 tablet_bootstrap.cc:492] T dfd1d70561924773aa204b5f22270b2e P ddb0f390a80f4c059c1a99a167c7d830: No bootstrap required, opened a new log
I20240502 20:47:00.364478 26593 ts_tablet_manager.cc:1367] T dfd1d70561924773aa204b5f22270b2e P ddb0f390a80f4c059c1a99a167c7d830: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 20:47:00.364558 26598 leader_election.cc:288] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [CANDIDATE]: Term 1 election: Requested vote from peers 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989), ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519)
I20240502 20:47:00.364804 26391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1409e44c98f44bd888eea4cee4f1163a" candidate_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97406d5f0d1f4d59bdc784684baa1559"
I20240502 20:47:00.364830 26467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1409e44c98f44bd888eea4cee4f1163a" candidate_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830"
I20240502 20:47:00.364890 26391 raft_consensus.cc:3037] T 1409e44c98f44bd888eea4cee4f1163a P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.364907 26467 raft_consensus.cc:3037] T 1409e44c98f44bd888eea4cee4f1163a P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.364915 26593 raft_consensus.cc:348] T dfd1d70561924773aa204b5f22270b2e P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.364997 26593 raft_consensus.cc:374] T dfd1d70561924773aa204b5f22270b2e P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.365028 26593 raft_consensus.cc:729] T dfd1d70561924773aa204b5f22270b2e P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb0f390a80f4c059c1a99a167c7d830, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.365167 26593 consensus_queue.cc:260] T dfd1d70561924773aa204b5f22270b2e P ddb0f390a80f4c059c1a99a167c7d830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.365543 26467 raft_consensus.cc:2454] T 1409e44c98f44bd888eea4cee4f1163a P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 in term 1.
I20240502 20:47:00.365785 26505 leader_election.cc:302] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, ddb0f390a80f4c059c1a99a167c7d830; no voters:
I20240502 20:47:00.365831 26391 raft_consensus.cc:2454] T 1409e44c98f44bd888eea4cee4f1163a P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 in term 1.
I20240502 20:47:00.365918 26598 raft_consensus.cc:2781] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:47:00.366092 26598 raft_consensus.cc:686] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 LEADER]: Becoming Leader. State: Replica: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, State: Running, Role: LEADER
I20240502 20:47:00.366209 26598 consensus_queue.cc:237] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.366793 26593 ts_tablet_manager.cc:1398] T dfd1d70561924773aa204b5f22270b2e P ddb0f390a80f4c059c1a99a167c7d830: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:47:00.367481 26593 tablet_bootstrap.cc:492] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830: Bootstrap starting.
I20240502 20:47:00.367795 26286 catalog_manager.cc:5484] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131). New cstate: current_term: 1 leader_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: HEALTHY } } }
I20240502 20:47:00.368559 26593 tablet_bootstrap.cc:654] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.369769 26593 tablet_bootstrap.cc:492] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830: No bootstrap required, opened a new log
I20240502 20:47:00.369845 26593 ts_tablet_manager.cc:1367] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:47:00.370190 26593 raft_consensus.cc:348] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } }
I20240502 20:47:00.370273 26593 raft_consensus.cc:374] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.370304 26593 raft_consensus.cc:729] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb0f390a80f4c059c1a99a167c7d830, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.370424 26593 consensus_queue.cc:260] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } }
I20240502 20:47:00.370646 26593 ts_tablet_manager.cc:1398] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:47:00.370836 26593 tablet_bootstrap.cc:492] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830: Bootstrap starting.
I20240502 20:47:00.371712 26593 tablet_bootstrap.cc:654] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.372857 26593 tablet_bootstrap.cc:492] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830: No bootstrap required, opened a new log
I20240502 20:47:00.372951 26593 ts_tablet_manager.cc:1367] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:47:00.373193 26598 raft_consensus.cc:482] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:47:00.373286 26598 raft_consensus.cc:504] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.373358 26593 raft_consensus.cc:348] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.373479 26593 raft_consensus.cc:374] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.373513 26593 raft_consensus.cc:729] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb0f390a80f4c059c1a99a167c7d830, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.373627 26593 consensus_queue.cc:260] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.373692 26598 leader_election.cc:288] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989), ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519)
I20240502 20:47:00.373932 26391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082557bf92e4443cbb9d0c50a89d24c4" candidate_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97406d5f0d1f4d59bdc784684baa1559" is_pre_election: true
I20240502 20:47:00.373970 26467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082557bf92e4443cbb9d0c50a89d24c4" candidate_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830" is_pre_election: true
I20240502 20:47:00.374044 26391 raft_consensus.cc:2454] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 in term 0.
I20240502 20:47:00.374083 26467 raft_consensus.cc:2454] T 082557bf92e4443cbb9d0c50a89d24c4 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 in term 0.
I20240502 20:47:00.374104 26593 ts_tablet_manager.cc:1398] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:47:00.374262 26593 tablet_bootstrap.cc:492] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830: Bootstrap starting.
I20240502 20:47:00.374312 26507 leader_election.cc:302] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97406d5f0d1f4d59bdc784684baa1559, bdcd7bbd5fa14f21bbf0a7a3cdc33cb3; no voters:
I20240502 20:47:00.374457 26598 raft_consensus.cc:2781] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:47:00.374521 26598 raft_consensus.cc:482] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:47:00.374557 26598 raft_consensus.cc:3037] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.375418 26593 tablet_bootstrap.cc:654] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830: Neither blocks nor log segments found. Creating new log.
I20240502 20:47:00.375568 26598 raft_consensus.cc:504] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.375823 26635 raft_consensus.cc:482] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:47:00.375908 26598 leader_election.cc:288] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [CANDIDATE]: Term 1 election: Requested vote from peers 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989), ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519)
I20240502 20:47:00.375906 26635 raft_consensus.cc:504] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.376206 26635 leader_election.cc:288] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989), ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519)
I20240502 20:47:00.376577 26391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082557bf92e4443cbb9d0c50a89d24c4" candidate_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97406d5f0d1f4d59bdc784684baa1559"
I20240502 20:47:00.376683 26391 raft_consensus.cc:3037] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.376711 26467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082557bf92e4443cbb9d0c50a89d24c4" candidate_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830"
I20240502 20:47:00.376716 26390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0df961aea849f4bc2ea340349f8848" candidate_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97406d5f0d1f4d59bdc784684baa1559" is_pre_election: true
I20240502 20:47:00.376824 26467 raft_consensus.cc:3037] T 082557bf92e4443cbb9d0c50a89d24c4 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.376861 26390 raft_consensus.cc:2454] T ca0df961aea849f4bc2ea340349f8848 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 in term 0.
I20240502 20:47:00.377442 26391 raft_consensus.cc:2454] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 in term 1.
I20240502 20:47:00.377511 26467 raft_consensus.cc:2454] T 082557bf92e4443cbb9d0c50a89d24c4 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 in term 1.
I20240502 20:47:00.377730 26466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0df961aea849f4bc2ea340349f8848" candidate_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830" is_pre_election: true
I20240502 20:47:00.377770 26505 leader_election.cc:302] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, ddb0f390a80f4c059c1a99a167c7d830; no voters:
I20240502 20:47:00.377842 26466 raft_consensus.cc:2454] T ca0df961aea849f4bc2ea340349f8848 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 in term 0.
I20240502 20:47:00.377905 26593 tablet_bootstrap.cc:492] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830: No bootstrap required, opened a new log
I20240502 20:47:00.377910 26635 raft_consensus.cc:2781] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:47:00.378019 26593 ts_tablet_manager.cc:1367] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:47:00.378142 26507 leader_election.cc:302] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97406d5f0d1f4d59bdc784684baa1559, bdcd7bbd5fa14f21bbf0a7a3cdc33cb3; no voters:
I20240502 20:47:00.378048 26635 raft_consensus.cc:686] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 LEADER]: Becoming Leader. State: Replica: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, State: Running, Role: LEADER
I20240502 20:47:00.378453 26635 consensus_queue.cc:237] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.378471 26593 raft_consensus.cc:348] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.378544 26593 raft_consensus.cc:374] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:47:00.378562 26593 raft_consensus.cc:729] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb0f390a80f4c059c1a99a167c7d830, State: Initialized, Role: FOLLOWER
I20240502 20:47:00.378671 26593 consensus_queue.cc:260] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.378765 26598 raft_consensus.cc:2781] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:47:00.378827 26598 raft_consensus.cc:482] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:47:00.378906 26598 raft_consensus.cc:3037] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.379024 26593 ts_tablet_manager.cc:1398] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:47:00.379719 26598 raft_consensus.cc:504] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.380059 26597 raft_consensus.cc:482] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:47:00.380134 26597 raft_consensus.cc:504] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.380179 26598 leader_election.cc:288] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [CANDIDATE]: Term 1 election: Requested vote from peers 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989), ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519)
I20240502 20:47:00.380419 26391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0df961aea849f4bc2ea340349f8848" candidate_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97406d5f0d1f4d59bdc784684baa1559"
I20240502 20:47:00.380514 26391 raft_consensus.cc:3037] T ca0df961aea849f4bc2ea340349f8848 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.380532 26597 leader_election.cc:288] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989), bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881)
I20240502 20:47:00.380667 26286 catalog_manager.cc:5484] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131). New cstate: current_term: 1 leader_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: HEALTHY } } }
I20240502 20:47:00.381175 26391 raft_consensus.cc:2454] T ca0df961aea849f4bc2ea340349f8848 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 in term 1.
I20240502 20:47:00.381790 26466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0df961aea849f4bc2ea340349f8848" candidate_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830"
I20240502 20:47:00.381788 26507 leader_election.cc:302] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97406d5f0d1f4d59bdc784684baa1559, bdcd7bbd5fa14f21bbf0a7a3cdc33cb3; no voters:
I20240502 20:47:00.381923 26466 raft_consensus.cc:3037] T ca0df961aea849f4bc2ea340349f8848 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.381997 26598 raft_consensus.cc:2781] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:47:00.382081 26598 raft_consensus.cc:686] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 LEADER]: Becoming Leader. State: Replica: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, State: Running, Role: LEADER
I20240502 20:47:00.382186 26598 consensus_queue.cc:237] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.382927 26466 raft_consensus.cc:2454] T ca0df961aea849f4bc2ea340349f8848 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 in term 1.
I20240502 20:47:00.385219 26286 catalog_manager.cc:5484] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131). New cstate: current_term: 1 leader_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: HEALTHY } } }
I20240502 20:47:00.385924 26391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5db41d5853a34dc594bafb0f4c53cd7d" candidate_uuid: "ddb0f390a80f4c059c1a99a167c7d830" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97406d5f0d1f4d59bdc784684baa1559" is_pre_election: true
I20240502 20:47:00.386058 26391 raft_consensus.cc:2454] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddb0f390a80f4c059c1a99a167c7d830 in term 0.
I20240502 20:47:00.386304 26432 leader_election.cc:302] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97406d5f0d1f4d59bdc784684baa1559, ddb0f390a80f4c059c1a99a167c7d830; no voters:
I20240502 20:47:00.386466 26597 raft_consensus.cc:2781] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:47:00.386533 26597 raft_consensus.cc:482] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:47:00.386579 26597 raft_consensus.cc:3037] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.387522 26597 raft_consensus.cc:504] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.387836 26597 leader_election.cc:288] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [CANDIDATE]: Term 1 election: Requested vote from peers 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989), bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881)
I20240502 20:47:00.388139 26391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5db41d5853a34dc594bafb0f4c53cd7d" candidate_uuid: "ddb0f390a80f4c059c1a99a167c7d830" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97406d5f0d1f4d59bdc784684baa1559"
I20240502 20:47:00.388242 26391 raft_consensus.cc:3037] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.388908 26391 raft_consensus.cc:2454] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddb0f390a80f4c059c1a99a167c7d830 in term 1.
I20240502 20:47:00.389250 26542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5db41d5853a34dc594bafb0f4c53cd7d" candidate_uuid: "ddb0f390a80f4c059c1a99a167c7d830" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" is_pre_election: true
I20240502 20:47:00.389318 26543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5db41d5853a34dc594bafb0f4c53cd7d" candidate_uuid: "ddb0f390a80f4c059c1a99a167c7d830" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3"
I20240502 20:47:00.389348 26542 raft_consensus.cc:2454] T 5db41d5853a34dc594bafb0f4c53cd7d P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddb0f390a80f4c059c1a99a167c7d830 in term 0.
I20240502 20:47:00.389377 26432 leader_election.cc:302] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97406d5f0d1f4d59bdc784684baa1559, ddb0f390a80f4c059c1a99a167c7d830; no voters:
I20240502 20:47:00.389855 26597 raft_consensus.cc:2781] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:47:00.390065 26597 raft_consensus.cc:686] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [term 1 LEADER]: Becoming Leader. State: Replica: ddb0f390a80f4c059c1a99a167c7d830, State: Running, Role: LEADER
I20240502 20:47:00.390210 26597 consensus_queue.cc:237] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.391808 26286 catalog_manager.cc:5484] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130). New cstate: current_term: 1 leader_uuid: "ddb0f390a80f4c059c1a99a167c7d830" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:00.399094 26597 raft_consensus.cc:482] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:47:00.399240 26597 raft_consensus.cc:504] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.399644 26597 leader_election.cc:288] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989), bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881)
I20240502 20:47:00.400053 26391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea4b2412a3a420f93d384aa48117c60" candidate_uuid: "ddb0f390a80f4c059c1a99a167c7d830" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97406d5f0d1f4d59bdc784684baa1559" is_pre_election: true
I20240502 20:47:00.400053 26543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea4b2412a3a420f93d384aa48117c60" candidate_uuid: "ddb0f390a80f4c059c1a99a167c7d830" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" is_pre_election: true
I20240502 20:47:00.400209 26391 raft_consensus.cc:2454] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddb0f390a80f4c059c1a99a167c7d830 in term 0.
I20240502 20:47:00.400244 26543 raft_consensus.cc:2454] T 8ea4b2412a3a420f93d384aa48117c60 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddb0f390a80f4c059c1a99a167c7d830 in term 0.
I20240502 20:47:00.400496 26432 leader_election.cc:302] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97406d5f0d1f4d59bdc784684baa1559, ddb0f390a80f4c059c1a99a167c7d830; no voters:
I20240502 20:47:00.400660 26597 raft_consensus.cc:2781] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:47:00.400719 26597 raft_consensus.cc:482] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:47:00.400764 26597 raft_consensus.cc:3037] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.401575 26597 raft_consensus.cc:504] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.401912 26597 leader_election.cc:288] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [CANDIDATE]: Term 1 election: Requested vote from peers 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989), bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881)
I20240502 20:47:00.402232 26391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea4b2412a3a420f93d384aa48117c60" candidate_uuid: "ddb0f390a80f4c059c1a99a167c7d830" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97406d5f0d1f4d59bdc784684baa1559"
I20240502 20:47:00.402350 26391 raft_consensus.cc:3037] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.402575 26543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea4b2412a3a420f93d384aa48117c60" candidate_uuid: "ddb0f390a80f4c059c1a99a167c7d830" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3"
I20240502 20:47:00.402675 26543 raft_consensus.cc:3037] T 8ea4b2412a3a420f93d384aa48117c60 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.403518 26391 raft_consensus.cc:2454] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddb0f390a80f4c059c1a99a167c7d830 in term 1.
I20240502 20:47:00.403685 26543 raft_consensus.cc:2454] T 8ea4b2412a3a420f93d384aa48117c60 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddb0f390a80f4c059c1a99a167c7d830 in term 1.
I20240502 20:47:00.403820 26432 leader_election.cc:302] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97406d5f0d1f4d59bdc784684baa1559, ddb0f390a80f4c059c1a99a167c7d830; no voters:
I20240502 20:47:00.403983 26597 raft_consensus.cc:2781] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:47:00.404073 26597 raft_consensus.cc:686] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 LEADER]: Becoming Leader. State: Replica: ddb0f390a80f4c059c1a99a167c7d830, State: Running, Role: LEADER
I20240502 20:47:00.404218 26597 consensus_queue.cc:237] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.405885 26286 catalog_manager.cc:5484] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130). New cstate: current_term: 1 leader_uuid: "ddb0f390a80f4c059c1a99a167c7d830" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:00.423817 26597 raft_consensus.cc:482] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:47:00.423974 26597 raft_consensus.cc:504] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } }
I20240502 20:47:00.424417 26597 leader_election.cc:288] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881), 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989)
I20240502 20:47:00.424898 26543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32631c89419245dea46abe6baa5dc295" candidate_uuid: "ddb0f390a80f4c059c1a99a167c7d830" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" is_pre_election: true
I20240502 20:47:00.424914 26391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32631c89419245dea46abe6baa5dc295" candidate_uuid: "ddb0f390a80f4c059c1a99a167c7d830" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97406d5f0d1f4d59bdc784684baa1559" is_pre_election: true
I20240502 20:47:00.425038 26543 raft_consensus.cc:2454] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddb0f390a80f4c059c1a99a167c7d830 in term 0.
I20240502 20:47:00.425076 26391 raft_consensus.cc:2454] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddb0f390a80f4c059c1a99a167c7d830 in term 0.
I20240502 20:47:00.425323 26431 leader_election.cc:302] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, ddb0f390a80f4c059c1a99a167c7d830; no voters:
I20240502 20:47:00.425499 26597 raft_consensus.cc:2781] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:47:00.425556 26597 raft_consensus.cc:482] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:47:00.425596 26597 raft_consensus.cc:3037] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.426360 26597 raft_consensus.cc:504] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } }
I20240502 20:47:00.426687 26597 leader_election.cc:288] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [CANDIDATE]: Term 1 election: Requested vote from peers bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881), 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989)
I20240502 20:47:00.427008 26543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32631c89419245dea46abe6baa5dc295" candidate_uuid: "ddb0f390a80f4c059c1a99a167c7d830" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3"
I20240502 20:47:00.427038 26391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32631c89419245dea46abe6baa5dc295" candidate_uuid: "ddb0f390a80f4c059c1a99a167c7d830" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97406d5f0d1f4d59bdc784684baa1559"
I20240502 20:47:00.427110 26543 raft_consensus.cc:3037] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.427129 26391 raft_consensus.cc:3037] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.427804 26543 raft_consensus.cc:2454] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddb0f390a80f4c059c1a99a167c7d830 in term 1.
I20240502 20:47:00.427812 26391 raft_consensus.cc:2454] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddb0f390a80f4c059c1a99a167c7d830 in term 1.
I20240502 20:47:00.428073 26431 leader_election.cc:302] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, ddb0f390a80f4c059c1a99a167c7d830; no voters:
I20240502 20:47:00.428223 26597 raft_consensus.cc:2781] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:47:00.428308 26597 raft_consensus.cc:686] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 LEADER]: Becoming Leader. State: Replica: ddb0f390a80f4c059c1a99a167c7d830, State: Running, Role: LEADER
I20240502 20:47:00.428442 26597 consensus_queue.cc:237] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } }
I20240502 20:47:00.430032 26286 catalog_manager.cc:5484] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130). New cstate: current_term: 1 leader_uuid: "ddb0f390a80f4c059c1a99a167c7d830" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:00.479576 26622 raft_consensus.cc:482] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:47:00.479733 26622 raft_consensus.cc:504] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.480185 26622 leader_election.cc:288] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519), bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881)
I20240502 20:47:00.480583 26466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "912fa9b5ccf74aadb519a8bf16659478" candidate_uuid: "97406d5f0d1f4d59bdc784684baa1559" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830" is_pre_election: true
I20240502 20:47:00.480720 26466 raft_consensus.cc:2454] T 912fa9b5ccf74aadb519a8bf16659478 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97406d5f0d1f4d59bdc784684baa1559 in term 0.
I20240502 20:47:00.480827 26543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "912fa9b5ccf74aadb519a8bf16659478" candidate_uuid: "97406d5f0d1f4d59bdc784684baa1559" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" is_pre_election: true
I20240502 20:47:00.480965 26353 leader_election.cc:302] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97406d5f0d1f4d59bdc784684baa1559, ddb0f390a80f4c059c1a99a167c7d830; no voters:
I20240502 20:47:00.480963 26543 raft_consensus.cc:2454] T 912fa9b5ccf74aadb519a8bf16659478 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97406d5f0d1f4d59bdc784684baa1559 in term 0.
I20240502 20:47:00.481168 26622 raft_consensus.cc:2781] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:47:00.481228 26622 raft_consensus.cc:482] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:47:00.481267 26622 raft_consensus.cc:3037] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.482126 26622 raft_consensus.cc:504] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.482424 26622 leader_election.cc:288] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [CANDIDATE]: Term 1 election: Requested vote from peers ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519), bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881)
I20240502 20:47:00.482756 26543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "912fa9b5ccf74aadb519a8bf16659478" candidate_uuid: "97406d5f0d1f4d59bdc784684baa1559" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3"
I20240502 20:47:00.482734 26466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "912fa9b5ccf74aadb519a8bf16659478" candidate_uuid: "97406d5f0d1f4d59bdc784684baa1559" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830"
I20240502 20:47:00.482849 26543 raft_consensus.cc:3037] T 912fa9b5ccf74aadb519a8bf16659478 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.482854 26466 raft_consensus.cc:3037] T 912fa9b5ccf74aadb519a8bf16659478 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.483700 26543 raft_consensus.cc:2454] T 912fa9b5ccf74aadb519a8bf16659478 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97406d5f0d1f4d59bdc784684baa1559 in term 1.
I20240502 20:47:00.483700 26466 raft_consensus.cc:2454] T 912fa9b5ccf74aadb519a8bf16659478 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97406d5f0d1f4d59bdc784684baa1559 in term 1.
I20240502 20:47:00.483987 26354 leader_election.cc:302] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97406d5f0d1f4d59bdc784684baa1559, bdcd7bbd5fa14f21bbf0a7a3cdc33cb3; no voters:
I20240502 20:47:00.484131 26622 raft_consensus.cc:2781] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:47:00.484215 26622 raft_consensus.cc:686] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 LEADER]: Becoming Leader. State: Replica: 97406d5f0d1f4d59bdc784684baa1559, State: Running, Role: LEADER
I20240502 20:47:00.484364 26622 consensus_queue.cc:237] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.485913 26286 catalog_manager.cc:5484] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129). New cstate: current_term: 1 leader_uuid: "97406d5f0d1f4d59bdc784684baa1559" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:00.551385 26597 raft_consensus.cc:482] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:47:00.551657 26597 raft_consensus.cc:504] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.552167 26597 leader_election.cc:288] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989), bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881)
I20240502 20:47:00.552592 26543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1ff41afa6e46609e6a3eb277df18ad" candidate_uuid: "ddb0f390a80f4c059c1a99a167c7d830" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" is_pre_election: true
I20240502 20:47:00.552592 26391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1ff41afa6e46609e6a3eb277df18ad" candidate_uuid: "ddb0f390a80f4c059c1a99a167c7d830" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97406d5f0d1f4d59bdc784684baa1559" is_pre_election: true
I20240502 20:47:00.552719 26543 raft_consensus.cc:2454] T 7d1ff41afa6e46609e6a3eb277df18ad P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddb0f390a80f4c059c1a99a167c7d830 in term 0.
I20240502 20:47:00.552742 26391 raft_consensus.cc:2454] T 7d1ff41afa6e46609e6a3eb277df18ad P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddb0f390a80f4c059c1a99a167c7d830 in term 0.
I20240502 20:47:00.552949 26431 leader_election.cc:302] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, ddb0f390a80f4c059c1a99a167c7d830; no voters:
I20240502 20:47:00.553098 26597 raft_consensus.cc:2781] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:47:00.553155 26597 raft_consensus.cc:482] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:47:00.553196 26597 raft_consensus.cc:3037] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.554003 26597 raft_consensus.cc:504] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.554291 26597 leader_election.cc:288] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [CANDIDATE]: Term 1 election: Requested vote from peers 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989), bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881)
I20240502 20:47:00.554551 26391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1ff41afa6e46609e6a3eb277df18ad" candidate_uuid: "ddb0f390a80f4c059c1a99a167c7d830" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97406d5f0d1f4d59bdc784684baa1559"
I20240502 20:47:00.554592 26543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1ff41afa6e46609e6a3eb277df18ad" candidate_uuid: "ddb0f390a80f4c059c1a99a167c7d830" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3"
I20240502 20:47:00.554656 26391 raft_consensus.cc:3037] T 7d1ff41afa6e46609e6a3eb277df18ad P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.554682 26543 raft_consensus.cc:3037] T 7d1ff41afa6e46609e6a3eb277df18ad P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.555377 26391 raft_consensus.cc:2454] T 7d1ff41afa6e46609e6a3eb277df18ad P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddb0f390a80f4c059c1a99a167c7d830 in term 1.
I20240502 20:47:00.555409 26543 raft_consensus.cc:2454] T 7d1ff41afa6e46609e6a3eb277df18ad P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddb0f390a80f4c059c1a99a167c7d830 in term 1.
I20240502 20:47:00.555629 26432 leader_election.cc:302] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97406d5f0d1f4d59bdc784684baa1559, ddb0f390a80f4c059c1a99a167c7d830; no voters:
I20240502 20:47:00.555799 26597 raft_consensus.cc:2781] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:47:00.555927 26597 raft_consensus.cc:686] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [term 1 LEADER]: Becoming Leader. State: Replica: ddb0f390a80f4c059c1a99a167c7d830, State: Running, Role: LEADER
I20240502 20:47:00.556061 26597 consensus_queue.cc:237] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.557672 26286 catalog_manager.cc:5484] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130). New cstate: current_term: 1 leader_uuid: "ddb0f390a80f4c059c1a99a167c7d830" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:00.562217 26622 raft_consensus.cc:482] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:47:00.562307 26622 raft_consensus.cc:504] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.562659 26622 leader_election.cc:288] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519), bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881)
I20240502 20:47:00.562987 26466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f60af679e74f078fa50af755a9fbf6" candidate_uuid: "97406d5f0d1f4d59bdc784684baa1559" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830" is_pre_election: true
I20240502 20:47:00.562987 26543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f60af679e74f078fa50af755a9fbf6" candidate_uuid: "97406d5f0d1f4d59bdc784684baa1559" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" is_pre_election: true
I20240502 20:47:00.563146 26466 raft_consensus.cc:2454] T 23f60af679e74f078fa50af755a9fbf6 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97406d5f0d1f4d59bdc784684baa1559 in term 0.
I20240502 20:47:00.563146 26543 raft_consensus.cc:2454] T 23f60af679e74f078fa50af755a9fbf6 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97406d5f0d1f4d59bdc784684baa1559 in term 0.
I20240502 20:47:00.563409 26354 leader_election.cc:302] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97406d5f0d1f4d59bdc784684baa1559, bdcd7bbd5fa14f21bbf0a7a3cdc33cb3; no voters:
I20240502 20:47:00.563562 26622 raft_consensus.cc:2781] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:47:00.563616 26622 raft_consensus.cc:482] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:47:00.563644 26622 raft_consensus.cc:3037] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.564428 26622 raft_consensus.cc:504] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.564736 26622 leader_election.cc:288] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559 [CANDIDATE]: Term 1 election: Requested vote from peers ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519), bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881)
I20240502 20:47:00.565035 26466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f60af679e74f078fa50af755a9fbf6" candidate_uuid: "97406d5f0d1f4d59bdc784684baa1559" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830"
I20240502 20:47:00.565040 26543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f60af679e74f078fa50af755a9fbf6" candidate_uuid: "97406d5f0d1f4d59bdc784684baa1559" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3"
I20240502 20:47:00.565140 26466 raft_consensus.cc:3037] T 23f60af679e74f078fa50af755a9fbf6 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.565146 26543 raft_consensus.cc:3037] T 23f60af679e74f078fa50af755a9fbf6 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.565819 26543 raft_consensus.cc:2454] T 23f60af679e74f078fa50af755a9fbf6 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97406d5f0d1f4d59bdc784684baa1559 in term 1.
I20240502 20:47:00.565855 26466 raft_consensus.cc:2454] T 23f60af679e74f078fa50af755a9fbf6 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97406d5f0d1f4d59bdc784684baa1559 in term 1.
I20240502 20:47:00.566058 26354 leader_election.cc:302] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97406d5f0d1f4d59bdc784684baa1559, bdcd7bbd5fa14f21bbf0a7a3cdc33cb3; no voters:
I20240502 20:47:00.566220 26622 raft_consensus.cc:2781] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:47:00.566305 26622 raft_consensus.cc:686] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 LEADER]: Becoming Leader. State: Replica: 97406d5f0d1f4d59bdc784684baa1559, State: Running, Role: LEADER
I20240502 20:47:00.566431 26622 consensus_queue.cc:237] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.568121 26286 catalog_manager.cc:5484] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129). New cstate: current_term: 1 leader_uuid: "97406d5f0d1f4d59bdc784684baa1559" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:00.596376 26598 raft_consensus.cc:482] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:47:00.596531 26598 raft_consensus.cc:504] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.596958 26598 leader_election.cc:288] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989), ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519)
I20240502 20:47:00.597355 26466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc72d9e28894957a82f3b3ef37e6f08" candidate_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830" is_pre_election: true
I20240502 20:47:00.597363 26391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc72d9e28894957a82f3b3ef37e6f08" candidate_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97406d5f0d1f4d59bdc784684baa1559" is_pre_election: true
I20240502 20:47:00.597505 26466 raft_consensus.cc:2454] T 1bc72d9e28894957a82f3b3ef37e6f08 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 in term 0.
I20240502 20:47:00.597532 26391 raft_consensus.cc:2454] T 1bc72d9e28894957a82f3b3ef37e6f08 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 in term 0.
I20240502 20:47:00.597779 26505 leader_election.cc:302] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, ddb0f390a80f4c059c1a99a167c7d830; no voters:
I20240502 20:47:00.597934 26598 raft_consensus.cc:2781] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:47:00.597992 26598 raft_consensus.cc:482] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:47:00.598037 26598 raft_consensus.cc:3037] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.598847 26598 raft_consensus.cc:504] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.599174 26598 leader_election.cc:288] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [CANDIDATE]: Term 1 election: Requested vote from peers 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989), ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519)
I20240502 20:47:00.599470 26391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc72d9e28894957a82f3b3ef37e6f08" candidate_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97406d5f0d1f4d59bdc784684baa1559"
I20240502 20:47:00.599510 26466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc72d9e28894957a82f3b3ef37e6f08" candidate_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830"
I20240502 20:47:00.599571 26391 raft_consensus.cc:3037] T 1bc72d9e28894957a82f3b3ef37e6f08 P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.599607 26466 raft_consensus.cc:3037] T 1bc72d9e28894957a82f3b3ef37e6f08 P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.600283 26391 raft_consensus.cc:2454] T 1bc72d9e28894957a82f3b3ef37e6f08 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 in term 1.
I20240502 20:47:00.600281 26466 raft_consensus.cc:2454] T 1bc72d9e28894957a82f3b3ef37e6f08 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 in term 1.
I20240502 20:47:00.600576 26507 leader_election.cc:302] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97406d5f0d1f4d59bdc784684baa1559, bdcd7bbd5fa14f21bbf0a7a3cdc33cb3; no voters:
I20240502 20:47:00.600741 26598 raft_consensus.cc:2781] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:47:00.600823 26598 raft_consensus.cc:686] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 LEADER]: Becoming Leader. State: Replica: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, State: Running, Role: LEADER
I20240502 20:47:00.600956 26598 consensus_queue.cc:237] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.602526 26286 catalog_manager.cc:5484] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131). New cstate: current_term: 1 leader_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: HEALTHY } } }
I20240502 20:47:00.683616 26622 raft_consensus.cc:482] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:47:00.683776 26622 raft_consensus.cc:504] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.684206 26622 leader_election.cc:288] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519), bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881)
I20240502 20:47:00.684689 26543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfd1d70561924773aa204b5f22270b2e" candidate_uuid: "97406d5f0d1f4d59bdc784684baa1559" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" is_pre_election: true
I20240502 20:47:00.684690 26466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfd1d70561924773aa204b5f22270b2e" candidate_uuid: "97406d5f0d1f4d59bdc784684baa1559" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830" is_pre_election: true
I20240502 20:47:00.684835 26543 raft_consensus.cc:2454] T dfd1d70561924773aa204b5f22270b2e P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97406d5f0d1f4d59bdc784684baa1559 in term 0.
I20240502 20:47:00.684862 26466 raft_consensus.cc:2454] T dfd1d70561924773aa204b5f22270b2e P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97406d5f0d1f4d59bdc784684baa1559 in term 0.
I20240502 20:47:00.685127 26353 leader_election.cc:302] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97406d5f0d1f4d59bdc784684baa1559, ddb0f390a80f4c059c1a99a167c7d830; no voters:
I20240502 20:47:00.685281 26622 raft_consensus.cc:2781] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:47:00.685339 26622 raft_consensus.cc:482] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:47:00.685379 26622 raft_consensus.cc:3037] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.686151 26622 raft_consensus.cc:504] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.686465 26622 leader_election.cc:288] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559 [CANDIDATE]: Term 1 election: Requested vote from peers ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519), bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881)
I20240502 20:47:00.686800 26466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfd1d70561924773aa204b5f22270b2e" candidate_uuid: "97406d5f0d1f4d59bdc784684baa1559" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830"
I20240502 20:47:00.686800 26543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfd1d70561924773aa204b5f22270b2e" candidate_uuid: "97406d5f0d1f4d59bdc784684baa1559" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3"
I20240502 20:47:00.686934 26543 raft_consensus.cc:3037] T dfd1d70561924773aa204b5f22270b2e P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.686933 26466 raft_consensus.cc:3037] T dfd1d70561924773aa204b5f22270b2e P ddb0f390a80f4c059c1a99a167c7d830 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.687831 26466 raft_consensus.cc:2454] T dfd1d70561924773aa204b5f22270b2e P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97406d5f0d1f4d59bdc784684baa1559 in term 1.
I20240502 20:47:00.687831 26543 raft_consensus.cc:2454] T dfd1d70561924773aa204b5f22270b2e P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97406d5f0d1f4d59bdc784684baa1559 in term 1.
I20240502 20:47:00.688151 26354 leader_election.cc:302] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 97406d5f0d1f4d59bdc784684baa1559, bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, ddb0f390a80f4c059c1a99a167c7d830; no voters:
I20240502 20:47:00.688310 26622 raft_consensus.cc:2781] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:47:00.688388 26622 raft_consensus.cc:686] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559 [term 1 LEADER]: Becoming Leader. State: Replica: 97406d5f0d1f4d59bdc784684baa1559, State: Running, Role: LEADER
I20240502 20:47:00.688524 26622 consensus_queue.cc:237] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:00.690234 26286 catalog_manager.cc:5484] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129). New cstate: current_term: 1 leader_uuid: "97406d5f0d1f4d59bdc784684baa1559" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:00.766647 26543 raft_consensus.cc:3037] T 5db41d5853a34dc594bafb0f4c53cd7d P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:47:00.768314 26597 consensus_queue.cc:1035] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:47:00.771661 26644 consensus_queue.cc:1035] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:47:00.781457 26466 raft_consensus.cc:1261] T 23f60af679e74f078fa50af755a9fbf6 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:47:00.782158 26466 raft_consensus.cc:1261] T dfd1d70561924773aa204b5f22270b2e P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:47:00.782626 26671 consensus_queue.cc:1035] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:47:00.783035 26671 consensus_queue.cc:1035] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:47:00.783072 26467 raft_consensus.cc:1261] T 912fa9b5ccf74aadb519a8bf16659478 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:47:00.783432 26466 raft_consensus.cc:1261] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:47:00.783975 26543 raft_consensus.cc:1261] T 23f60af679e74f078fa50af755a9fbf6 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:47:00.784257 26542 raft_consensus.cc:1261] T dfd1d70561924773aa204b5f22270b2e P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:47:00.784565 26541 raft_consensus.cc:1261] T 7d7f28e11e1f42959251a7b03011f5dc P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:47:00.784853 26540 raft_consensus.cc:1261] T 912fa9b5ccf74aadb519a8bf16659478 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:47:00.786821 26622 consensus_queue.cc:1035] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:47:00.787060 26622 consensus_queue.cc:1035] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:47:00.786829 26540 raft_consensus.cc:1261] T 7d1ff41afa6e46609e6a3eb277df18ad P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:47:00.787837 26644 consensus_queue.cc:1035] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:47:00.788247 26592 consensus_queue.cc:1035] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:47:00.788507 26592 consensus_queue.cc:1035] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:47:00.789366 26390 raft_consensus.cc:1261] T ca0df961aea849f4bc2ea340349f8848 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:47:00.789392 26391 raft_consensus.cc:1261] T 7d1ff41afa6e46609e6a3eb277df18ad P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:47:00.789577 26625 consensus_queue.cc:1035] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:47:00.789690 26597 consensus_queue.cc:1035] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:47:00.789798 26467 raft_consensus.cc:1261] T ca0df961aea849f4bc2ea340349f8848 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:47:00.790789 26635 consensus_queue.cc:1035] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:47:00.791749 26467 raft_consensus.cc:1261] T 082557bf92e4443cbb9d0c50a89d24c4 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:47:00.791898 26465 raft_consensus.cc:1261] T 1409e44c98f44bd888eea4cee4f1163a P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:47:00.792223 26464 raft_consensus.cc:1261] T 1bc72d9e28894957a82f3b3ef37e6f08 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:47:00.794183 26598 consensus_queue.cc:1035] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:47:00.794430 26598 consensus_queue.cc:1035] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:47:00.794644 26598 consensus_queue.cc:1035] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:47:00.795152 26622 consensus_queue.cc:1035] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:47:00.796269 26389 raft_consensus.cc:1261] T 1409e44c98f44bd888eea4cee4f1163a P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:47:00.796298 26390 raft_consensus.cc:1261] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:47:00.796554 26388 raft_consensus.cc:1261] T 1bc72d9e28894957a82f3b3ef37e6f08 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:47:00.798277 26388 raft_consensus.cc:1261] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:47:00.798611 26597 consensus_queue.cc:1035] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:47:00.798856 26635 consensus_queue.cc:1035] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:47:00.800884 26539 raft_consensus.cc:1261] T 8ea4b2412a3a420f93d384aa48117c60 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:47:00.803026 26635 consensus_queue.cc:1035] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:47:00.803319 26635 consensus_queue.cc:1035] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:47:00.803566 26635 consensus_queue.cc:1035] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:47:00.804229 26597 consensus_queue.cc:1035] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:47:00.804361 26539 raft_consensus.cc:1261] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:47:00.804435 26389 raft_consensus.cc:1261] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:47:00.806282 26644 consensus_queue.cc:1035] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:47:00.807299 26597 consensus_queue.cc:1035] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:47:00.817816 26659 mvcc.cc:205] Tried to move back new op lower bound from 7023340833924722688 to 7023340833041784832. Current Snapshot: MvccSnapshot[applied={T|T < 7023340833924722688}]
I20240502 20:47:00.832825 26706 mvcc.cc:205] Tried to move back new op lower bound from 7023340833964765184 to 7023340832271683584. Current Snapshot: MvccSnapshot[applied={T|T < 7023340833947652096}]
I20240502 20:47:00.861444 26709 mvcc.cc:205] Tried to move back new op lower bound from 7023340833967493120 to 7023340833182969856. Current Snapshot: MvccSnapshot[applied={T|T < 7023340833951678464}]
I20240502 20:47:01.098094 26341 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:47:01.099725 26341 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 20:47:01.099836 26341 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:47:01.293951 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:47:01.295879 26742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:47:01.297071 22946 server_base.cc:1034] running on GCE node
W20240502 20:47:01.299438 26741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:47:01.300227 26744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:47:01.300510 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:47:01.300561 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:47:01.300581 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682821300581 us; error 0 us; skew 500 ppm
I20240502 20:47:01.301703 22946 webserver.cc:460] Webserver started at http://127.22.104.132:32827/ using document root <none> and password file <none>
I20240502 20:47:01.301847 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:47:01.301895 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:47:01.301968 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:47:01.302323 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-3-root/instance:
uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f"
format_stamp: "Formatted at 2024-05-02 20:47:01 on dist-test-slave-rgns"
I20240502 20:47:01.303738 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 20:47:01.304561 26749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:47:01.304769 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 20:47:01.304831 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-3-root
uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f"
format_stamp: "Formatted at 2024-05-02 20:47:01 on dist-test-slave-rgns"
I20240502 20:47:01.304896 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:47:01.310895 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:47:01.311232 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:47:01.311483 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:47:01.311914 22946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:47:01.311951 22946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:47:01.311992 22946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:47:01.312018 22946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:47:01.316876 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.132:37635
I20240502 20:47:01.316973 26812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.132:37635 every 8 connection(s)
I20240502 20:47:01.317761 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:47:01.319581 26817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:47:01.320188 26820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:47:01.321098 22946 server_base.cc:1034] running on GCE node
W20240502 20:47:01.321087 26818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:47:01.321411 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:47:01.321465 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:47:01.321484 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682821321484 us; error 0 us; skew 500 ppm
I20240502 20:47:01.322034 26813 heartbeater.cc:347] Connected to a master server at 127.22.104.190:39193
I20240502 20:47:01.322127 26813 heartbeater.cc:464] Registering TS with master...
I20240502 20:47:01.322337 26813 heartbeater.cc:511] Master 127.22.104.190:39193 requested a full tablet report, sending...
I20240502 20:47:01.322525 22946 webserver.cc:460] Webserver started at http://127.22.104.133:46601/ using document root <none> and password file <none>
I20240502 20:47:01.322678 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:47:01.322760 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:47:01.322837 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:47:01.322978 26279 ts_manager.cc:194] Registered new tserver with Master: ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132:37635)
I20240502 20:47:01.323282 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-4-root/instance:
uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec"
format_stamp: "Formatted at 2024-05-02 20:47:01 on dist-test-slave-rgns"
I20240502 20:47:01.323675 26279 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58808
I20240502 20:47:01.324750 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 20:47:01.325559 26825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:47:01.325809 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:47:01.325872 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-4-root
uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec"
format_stamp: "Formatted at 2024-05-02 20:47:01 on dist-test-slave-rgns"
I20240502 20:47:01.325935 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-4-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-4-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-4-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:47:01.338069 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:47:01.338624 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:47:01.339066 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:47:01.339679 22946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:47:01.339731 22946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:47:01.339784 22946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:47:01.339812 22946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:47:01.346168 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.133:43257
I20240502 20:47:01.346333 26888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.133:43257 every 8 connection(s)
I20240502 20:47:01.347282 22946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:47:01.349331 26896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:47:01.349870 22946 server_base.cc:1034] running on GCE node
W20240502 20:47:01.350507 26894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:47:01.350751 26893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:47:01.351027 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:47:01.351079 22946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:47:01.351112 22946 hybrid_clock.cc:648] HybridClock initialized: now 1714682821351112 us; error 0 us; skew 500 ppm
I20240502 20:47:01.352062 26889 heartbeater.cc:347] Connected to a master server at 127.22.104.190:39193
I20240502 20:47:01.352169 26889 heartbeater.cc:464] Registering TS with master...
I20240502 20:47:01.352221 22946 webserver.cc:460] Webserver started at http://127.22.104.134:42937/ using document root <none> and password file <none>
I20240502 20:47:01.352375 22946 fs_manager.cc:359] Metadata directory not provided
I20240502 20:47:01.352377 26889 heartbeater.cc:511] Master 127.22.104.190:39193 requested a full tablet report, sending...
I20240502 20:47:01.352427 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:47:01.352516 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:47:01.352943 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-5-root/instance:
uuid: "c8ef2de5cb854720b1090ff20c2fd13c"
format_stamp: "Formatted at 2024-05-02 20:47:01 on dist-test-slave-rgns"
I20240502 20:47:01.352993 26279 ts_manager.cc:194] Registered new tserver with Master: 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133:43257)
I20240502 20:47:01.353502 26279 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58824
I20240502 20:47:01.354494 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20240502 20:47:01.355911 26901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:47:01.356184 22946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 20:47:01.356261 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-5-root
uuid: "c8ef2de5cb854720b1090ff20c2fd13c"
format_stamp: "Formatted at 2024-05-02 20:47:01 on dist-test-slave-rgns"
I20240502 20:47:01.356364 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-5-root
metadata directory: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-5-root
1 data directories: /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-5-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:47:01.361570 22946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:47:01.361960 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:47:01.362278 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:47:01.362879 22946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:47:01.362936 22946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:47:01.362979 22946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:47:01.363011 22946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:47:01.369112 22946 rpc_server.cc:307] RPC server started. Bound to: 127.22.104.134:45157
I20240502 20:47:01.369282 26965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.104.134:45157 every 8 connection(s)
I20240502 20:47:01.374341 26966 heartbeater.cc:347] Connected to a master server at 127.22.104.190:39193
I20240502 20:47:01.374455 26966 heartbeater.cc:464] Registering TS with master...
I20240502 20:47:01.374760 26966 heartbeater.cc:511] Master 127.22.104.190:39193 requested a full tablet report, sending...
I20240502 20:47:01.375368 26279 ts_manager.cc:194] Registered new tserver with Master: c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134:45157)
I20240502 20:47:01.375891 26279 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58836
I20240502 20:47:02.105176 26390 consensus_queue.cc:237] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } }
I20240502 20:47:02.106774 26538 raft_consensus.cc:1261] T 912fa9b5ccf74aadb519a8bf16659478 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 20:47:02.106930 26461 raft_consensus.cc:1261] T 912fa9b5ccf74aadb519a8bf16659478 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 20:47:02.107188 26972 consensus_queue.cc:1035] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 20:47:02.107414 26971 consensus_queue.cc:1035] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 20:47:02.108980 26972 raft_consensus.cc:2932] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } } }
I20240502 20:47:02.109453 26538 raft_consensus.cc:2932] T 912fa9b5ccf74aadb519a8bf16659478 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } } }
I20240502 20:47:02.109712 26461 raft_consensus.cc:2932] T 912fa9b5ccf74aadb519a8bf16659478 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } } }
W20240502 20:47:02.112365 26356 consensus_peers.cc:485] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 -> Peer ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132:37635): Couldn't send request to peer ae30e3bd7dc1421998cf9258aaa6bf7f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 912fa9b5ccf74aadb519a8bf16659478. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:02.112294 26279 catalog_manager.cc:5484] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 reported cstate change: config changed from index -1 to 27, NON_VOTER ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) added. New cstate: current_term: 1 leader_uuid: "97406d5f0d1f4d59bdc784684baa1559" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:02.116176 26538 consensus_queue.cc:237] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } }
I20240502 20:47:02.117527 26461 raft_consensus.cc:1261] T 1bc72d9e28894957a82f3b3ef37e6f08 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 20:47:02.117585 26390 raft_consensus.cc:1261] T 1bc72d9e28894957a82f3b3ef37e6f08 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 20:47:02.117914 26980 consensus_queue.cc:1035] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 20:47:02.118144 26981 consensus_queue.cc:1035] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 20:47:02.119622 26980 raft_consensus.cc:2932] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } } }
I20240502 20:47:02.120290 26464 raft_consensus.cc:2932] T 1bc72d9e28894957a82f3b3ef37e6f08 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } } }
I20240502 20:47:02.120339 26390 raft_consensus.cc:2932] T 1bc72d9e28894957a82f3b3ef37e6f08 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } } }
I20240502 20:47:02.122849 26281 catalog_manager.cc:5484] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 reported cstate change: config changed from index -1 to 29, NON_VOTER c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) added. New cstate: current_term: 1 leader_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 20:47:02.124225 26507 consensus_peers.cc:485] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 -> Peer c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134:45157): Couldn't send request to peer c8ef2de5cb854720b1090ff20c2fd13c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1bc72d9e28894957a82f3b3ef37e6f08. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:02.126237 26464 consensus_queue.cc:237] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } }
I20240502 20:47:02.127529 26390 raft_consensus.cc:1261] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 20:47:02.127615 26538 raft_consensus.cc:1261] T 8ea4b2412a3a420f93d384aa48117c60 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 20:47:02.127903 26739 consensus_queue.cc:1035] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 20:47:02.128127 26985 consensus_queue.cc:1035] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 20:47:02.129451 26739 raft_consensus.cc:2932] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } } }
I20240502 20:47:02.130101 26390 raft_consensus.cc:2932] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } } }
W20240502 20:47:02.131776 26430 consensus_peers.cc:485] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 -> Peer 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133:43257): Couldn't send request to peer 2c2b4bb6fbe94bb8bea2eb59520f5fec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ea4b2412a3a420f93d384aa48117c60. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:02.132027 26285 catalog_manager.cc:5484] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 reported cstate change: config changed from index -1 to 28, NON_VOTER 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) added. New cstate: current_term: 1 leader_uuid: "ddb0f390a80f4c059c1a99a167c7d830" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:02.132624 26538 raft_consensus.cc:2932] T 8ea4b2412a3a420f93d384aa48117c60 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } } }
I20240502 20:47:02.135094 26464 consensus_queue.cc:237] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } }
I20240502 20:47:02.136251 26538 raft_consensus.cc:1261] T 7d1ff41afa6e46609e6a3eb277df18ad P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 20:47:02.136327 26390 raft_consensus.cc:1261] T 7d1ff41afa6e46609e6a3eb277df18ad P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
W20240502 20:47:02.136520 26430 consensus_peers.cc:485] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 -> Peer 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133:43257): Couldn't send request to peer 2c2b4bb6fbe94bb8bea2eb59520f5fec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d1ff41afa6e46609e6a3eb277df18ad. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:02.136673 26988 consensus_queue.cc:1035] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 20:47:02.136850 26739 consensus_queue.cc:1035] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 20:47:02.138129 26985 raft_consensus.cc:2932] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } } }
I20240502 20:47:02.138501 26390 raft_consensus.cc:2932] T 7d1ff41afa6e46609e6a3eb277df18ad P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } } }
I20240502 20:47:02.138501 26538 raft_consensus.cc:2932] T 7d1ff41afa6e46609e6a3eb277df18ad P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } } }
I20240502 20:47:02.140697 26285 catalog_manager.cc:5484] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 reported cstate change: config changed from index -1 to 29, NON_VOTER 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) added. New cstate: current_term: 1 leader_uuid: "ddb0f390a80f4c059c1a99a167c7d830" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:02.141026 26538 consensus_queue.cc:237] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } }
I20240502 20:47:02.142226 26390 raft_consensus.cc:1261] T 1409e44c98f44bd888eea4cee4f1163a P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
W20240502 20:47:02.142459 26507 consensus_peers.cc:485] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 -> Peer c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134:45157): Couldn't send request to peer c8ef2de5cb854720b1090ff20c2fd13c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1409e44c98f44bd888eea4cee4f1163a. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:02.142645 26981 consensus_queue.cc:1035] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 20:47:02.142865 26464 raft_consensus.cc:1261] T 1409e44c98f44bd888eea4cee4f1163a P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 20:47:02.143179 26980 consensus_queue.cc:1035] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 20:47:02.144513 26678 raft_consensus.cc:2932] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } } }
I20240502 20:47:02.144840 26464 raft_consensus.cc:2932] T 1409e44c98f44bd888eea4cee4f1163a P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } } }
I20240502 20:47:02.144848 26390 raft_consensus.cc:2932] T 1409e44c98f44bd888eea4cee4f1163a P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } } }
I20240502 20:47:02.146888 26464 consensus_queue.cc:237] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } }
I20240502 20:47:02.147118 26281 catalog_manager.cc:5484] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 reported cstate change: config changed from index -1 to 29, NON_VOTER c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) added. New cstate: current_term: 1 leader_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:02.148587 26390 raft_consensus.cc:1261] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 20:47:02.148588 26538 raft_consensus.cc:1261] T 5db41d5853a34dc594bafb0f4c53cd7d P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 20:47:02.149046 26988 consensus_queue.cc:1035] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 20:47:02.149399 26988 consensus_queue.cc:1035] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 20:47:02.150691 26739 raft_consensus.cc:2932] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } } }
W20240502 20:47:02.151679 26433 consensus_peers.cc:485] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 -> Peer ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132:37635): Couldn't send request to peer ae30e3bd7dc1421998cf9258aaa6bf7f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5db41d5853a34dc594bafb0f4c53cd7d. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:02.151572 26391 raft_consensus.cc:2932] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } } }
I20240502 20:47:02.152650 26538 raft_consensus.cc:2932] T 5db41d5853a34dc594bafb0f4c53cd7d P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } } }
I20240502 20:47:02.153894 26279 catalog_manager.cc:5484] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 reported cstate change: config changed from index -1 to 29, NON_VOTER ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) added. New cstate: current_term: 1 leader_uuid: "ddb0f390a80f4c059c1a99a167c7d830" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:02.324522 26813 heartbeater.cc:503] Master 127.22.104.190:39193 was elected leader, sending a full tablet report...
I20240502 20:47:02.354282 26889 heartbeater.cc:503] Master 127.22.104.190:39193 was elected leader, sending a full tablet report...
I20240502 20:47:02.376660 26966 heartbeater.cc:503] Master 127.22.104.190:39193 was elected leader, sending a full tablet report...
I20240502 20:47:02.564247 26991 ts_tablet_manager.cc:902] T 1409e44c98f44bd888eea4cee4f1163a P c8ef2de5cb854720b1090ff20c2fd13c: Initiating tablet copy from peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881)
I20240502 20:47:02.564806 26991 tablet_copy_client.cc:308] T 1409e44c98f44bd888eea4cee4f1163a P c8ef2de5cb854720b1090ff20c2fd13c: tablet copy: Beginning tablet copy session from remote peer at address 127.22.104.131:38881
I20240502 20:47:02.568792 26553 tablet_copy_service.cc:140] P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Received BeginTabletCopySession request for tablet 1409e44c98f44bd888eea4cee4f1163a from peer c8ef2de5cb854720b1090ff20c2fd13c ({username='slave'} at 127.0.0.1:49944)
I20240502 20:47:02.568907 26553 tablet_copy_service.cc:161] P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Beginning new tablet copy session on tablet 1409e44c98f44bd888eea4cee4f1163a from peer c8ef2de5cb854720b1090ff20c2fd13c at {username='slave'} at 127.0.0.1:49944: session id = c8ef2de5cb854720b1090ff20c2fd13c-1409e44c98f44bd888eea4cee4f1163a
I20240502 20:47:02.569886 26553 tablet_copy_source_session.cc:205] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:47:02.570673 26997 ts_tablet_manager.cc:902] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec: Initiating tablet copy from peer ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519)
I20240502 20:47:02.570753 26991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1409e44c98f44bd888eea4cee4f1163a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:02.571235 26997 tablet_copy_client.cc:308] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec: tablet copy: Beginning tablet copy session from remote peer at address 127.22.104.130:35519
I20240502 20:47:02.572662 26991 tablet_copy_client.cc:783] T 1409e44c98f44bd888eea4cee4f1163a P c8ef2de5cb854720b1090ff20c2fd13c: tablet copy: Starting download of 0 data blocks...
I20240502 20:47:02.572803 26991 tablet_copy_client.cc:647] T 1409e44c98f44bd888eea4cee4f1163a P c8ef2de5cb854720b1090ff20c2fd13c: tablet copy: Starting download of 1 WAL segments...
I20240502 20:47:02.573755 26993 tablet_copy_client.cc:1002] T 1409e44c98f44bd888eea4cee4f1163a P c8ef2de5cb854720b1090ff20c2fd13c: tablet copy: Injecting latency into file download: 2000
I20240502 20:47:02.575034 26477 tablet_copy_service.cc:140] P ddb0f390a80f4c059c1a99a167c7d830: Received BeginTabletCopySession request for tablet 8ea4b2412a3a420f93d384aa48117c60 from peer 2c2b4bb6fbe94bb8bea2eb59520f5fec ({username='slave'} at 127.0.0.1:56918)
I20240502 20:47:02.575129 26477 tablet_copy_service.cc:161] P ddb0f390a80f4c059c1a99a167c7d830: Beginning new tablet copy session on tablet 8ea4b2412a3a420f93d384aa48117c60 from peer 2c2b4bb6fbe94bb8bea2eb59520f5fec at {username='slave'} at 127.0.0.1:56918: session id = 2c2b4bb6fbe94bb8bea2eb59520f5fec-8ea4b2412a3a420f93d384aa48117c60
I20240502 20:47:02.575958 26477 tablet_copy_source_session.cc:205] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:47:02.576659 26997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea4b2412a3a420f93d384aa48117c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:02.578366 26997 tablet_copy_client.cc:783] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec: tablet copy: Starting download of 0 data blocks...
I20240502 20:47:02.578511 26997 tablet_copy_client.cc:647] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec: tablet copy: Starting download of 1 WAL segments...
I20240502 20:47:02.579502 27001 tablet_copy_client.cc:1002] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec: tablet copy: Injecting latency into file download: 2000
I20240502 20:47:02.652215 27003 ts_tablet_manager.cc:902] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f: Initiating tablet copy from peer ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519)
I20240502 20:47:02.652694 27003 tablet_copy_client.cc:308] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f: tablet copy: Beginning tablet copy session from remote peer at address 127.22.104.130:35519
I20240502 20:47:02.656365 26477 tablet_copy_service.cc:140] P ddb0f390a80f4c059c1a99a167c7d830: Received BeginTabletCopySession request for tablet 5db41d5853a34dc594bafb0f4c53cd7d from peer ae30e3bd7dc1421998cf9258aaa6bf7f ({username='slave'} at 127.0.0.1:56924)
I20240502 20:47:02.656479 26477 tablet_copy_service.cc:161] P ddb0f390a80f4c059c1a99a167c7d830: Beginning new tablet copy session on tablet 5db41d5853a34dc594bafb0f4c53cd7d from peer ae30e3bd7dc1421998cf9258aaa6bf7f at {username='slave'} at 127.0.0.1:56924: session id = ae30e3bd7dc1421998cf9258aaa6bf7f-5db41d5853a34dc594bafb0f4c53cd7d
I20240502 20:47:02.657482 26477 tablet_copy_source_session.cc:205] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:47:02.658252 27003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5db41d5853a34dc594bafb0f4c53cd7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:02.660499 27003 tablet_copy_client.cc:783] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f: tablet copy: Starting download of 0 data blocks...
I20240502 20:47:02.660707 27003 tablet_copy_client.cc:647] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f: tablet copy: Starting download of 1 WAL segments...
I20240502 20:47:02.661600 27007 tablet_copy_client.cc:1002] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f: tablet copy: Injecting latency into file download: 2000
I20240502 20:47:02.692612 27009 ts_tablet_manager.cc:902] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c: Initiating tablet copy from peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881)
I20240502 20:47:02.693212 27009 tablet_copy_client.cc:308] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c: tablet copy: Beginning tablet copy session from remote peer at address 127.22.104.131:38881
I20240502 20:47:02.693712 26553 tablet_copy_service.cc:140] P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Received BeginTabletCopySession request for tablet 1bc72d9e28894957a82f3b3ef37e6f08 from peer c8ef2de5cb854720b1090ff20c2fd13c ({username='slave'} at 127.0.0.1:49944)
I20240502 20:47:02.693818 26553 tablet_copy_service.cc:161] P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Beginning new tablet copy session on tablet 1bc72d9e28894957a82f3b3ef37e6f08 from peer c8ef2de5cb854720b1090ff20c2fd13c at {username='slave'} at 127.0.0.1:49944: session id = c8ef2de5cb854720b1090ff20c2fd13c-1bc72d9e28894957a82f3b3ef37e6f08
I20240502 20:47:02.694672 26553 tablet_copy_source_session.cc:205] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:47:02.695436 27009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bc72d9e28894957a82f3b3ef37e6f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:02.697461 27009 tablet_copy_client.cc:783] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c: tablet copy: Starting download of 0 data blocks...
I20240502 20:47:02.697618 27009 tablet_copy_client.cc:647] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c: tablet copy: Starting download of 1 WAL segments...
I20240502 20:47:02.698662 27013 tablet_copy_client.cc:1002] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c: tablet copy: Injecting latency into file download: 2000
I20240502 20:47:02.718647 27015 ts_tablet_manager.cc:902] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f: Initiating tablet copy from peer 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989)
I20240502 20:47:02.719295 27015 tablet_copy_client.cc:308] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f: tablet copy: Beginning tablet copy session from remote peer at address 127.22.104.129:41989
I20240502 20:47:02.723028 26401 tablet_copy_service.cc:140] P 97406d5f0d1f4d59bdc784684baa1559: Received BeginTabletCopySession request for tablet 912fa9b5ccf74aadb519a8bf16659478 from peer ae30e3bd7dc1421998cf9258aaa6bf7f ({username='slave'} at 127.0.0.1:52196)
I20240502 20:47:02.723161 26401 tablet_copy_service.cc:161] P 97406d5f0d1f4d59bdc784684baa1559: Beginning new tablet copy session on tablet 912fa9b5ccf74aadb519a8bf16659478 from peer ae30e3bd7dc1421998cf9258aaa6bf7f at {username='slave'} at 127.0.0.1:52196: session id = ae30e3bd7dc1421998cf9258aaa6bf7f-912fa9b5ccf74aadb519a8bf16659478
I20240502 20:47:02.723942 26401 tablet_copy_source_session.cc:205] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:47:02.724833 27015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 912fa9b5ccf74aadb519a8bf16659478. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:02.727228 27015 tablet_copy_client.cc:783] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f: tablet copy: Starting download of 0 data blocks...
I20240502 20:47:02.727499 27015 tablet_copy_client.cc:647] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f: tablet copy: Starting download of 1 WAL segments...
I20240502 20:47:02.728368 27019 tablet_copy_client.cc:1002] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f: tablet copy: Injecting latency into file download: 2000
I20240502 20:47:02.747095 27021 ts_tablet_manager.cc:902] T 7d1ff41afa6e46609e6a3eb277df18ad P 2c2b4bb6fbe94bb8bea2eb59520f5fec: Initiating tablet copy from peer ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519)
I20240502 20:47:02.747735 27021 tablet_copy_client.cc:308] T 7d1ff41afa6e46609e6a3eb277df18ad P 2c2b4bb6fbe94bb8bea2eb59520f5fec: tablet copy: Beginning tablet copy session from remote peer at address 127.22.104.130:35519
I20240502 20:47:02.748117 26477 tablet_copy_service.cc:140] P ddb0f390a80f4c059c1a99a167c7d830: Received BeginTabletCopySession request for tablet 7d1ff41afa6e46609e6a3eb277df18ad from peer 2c2b4bb6fbe94bb8bea2eb59520f5fec ({username='slave'} at 127.0.0.1:56918)
I20240502 20:47:02.748230 26477 tablet_copy_service.cc:161] P ddb0f390a80f4c059c1a99a167c7d830: Beginning new tablet copy session on tablet 7d1ff41afa6e46609e6a3eb277df18ad from peer 2c2b4bb6fbe94bb8bea2eb59520f5fec at {username='slave'} at 127.0.0.1:56918: session id = 2c2b4bb6fbe94bb8bea2eb59520f5fec-7d1ff41afa6e46609e6a3eb277df18ad
I20240502 20:47:02.749159 26477 tablet_copy_source_session.cc:205] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:47:02.749907 27021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d1ff41afa6e46609e6a3eb277df18ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:02.752126 27021 tablet_copy_client.cc:783] T 7d1ff41afa6e46609e6a3eb277df18ad P 2c2b4bb6fbe94bb8bea2eb59520f5fec: tablet copy: Starting download of 0 data blocks...
I20240502 20:47:02.752271 27021 tablet_copy_client.cc:647] T 7d1ff41afa6e46609e6a3eb277df18ad P 2c2b4bb6fbe94bb8bea2eb59520f5fec: tablet copy: Starting download of 1 WAL segments...
I20240502 20:47:02.753398 27025 tablet_copy_client.cc:1002] T 7d1ff41afa6e46609e6a3eb277df18ad P 2c2b4bb6fbe94bb8bea2eb59520f5fec: tablet copy: Injecting latency into file download: 2000
I20240502 20:47:03.100087 26341 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:47:03.101614 26341 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 20:47:03.101748 26341 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:47:04.574081 26991 tablet_copy_client.cc:523] T 1409e44c98f44bd888eea4cee4f1163a P c8ef2de5cb854720b1090ff20c2fd13c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:47:04.578467 26991 tablet_bootstrap.cc:492] T 1409e44c98f44bd888eea4cee4f1163a P c8ef2de5cb854720b1090ff20c2fd13c: Bootstrap starting.
I20240502 20:47:04.579769 26997 tablet_copy_client.cc:523] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:47:04.581157 26997 tablet_bootstrap.cc:492] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec: Bootstrap starting.
I20240502 20:47:04.602094 26991 tablet_bootstrap.cc:492] T 1409e44c98f44bd888eea4cee4f1163a P c8ef2de5cb854720b1090ff20c2fd13c: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=107 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:47:04.602412 26991 tablet_bootstrap.cc:492] T 1409e44c98f44bd888eea4cee4f1163a P c8ef2de5cb854720b1090ff20c2fd13c: Bootstrap complete.
I20240502 20:47:04.602622 26991 ts_tablet_manager.cc:1367] T 1409e44c98f44bd888eea4cee4f1163a P c8ef2de5cb854720b1090ff20c2fd13c: Time spent bootstrapping tablet: real 0.024s user 0.019s sys 0.001s
I20240502 20:47:04.603099 26991 raft_consensus.cc:348] T 1409e44c98f44bd888eea4cee4f1163a P c8ef2de5cb854720b1090ff20c2fd13c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } }
I20240502 20:47:04.603241 26991 raft_consensus.cc:729] T 1409e44c98f44bd888eea4cee4f1163a P c8ef2de5cb854720b1090ff20c2fd13c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8ef2de5cb854720b1090ff20c2fd13c, State: Initialized, Role: LEARNER
I20240502 20:47:04.603376 26991 consensus_queue.cc:260] T 1409e44c98f44bd888eea4cee4f1163a P c8ef2de5cb854720b1090ff20c2fd13c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } }
I20240502 20:47:04.603732 26991 ts_tablet_manager.cc:1398] T 1409e44c98f44bd888eea4cee4f1163a P c8ef2de5cb854720b1090ff20c2fd13c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:47:04.603854 26997 tablet_bootstrap.cc:492] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=113 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:47:04.604125 26997 tablet_bootstrap.cc:492] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec: Bootstrap complete.
I20240502 20:47:04.604341 26997 ts_tablet_manager.cc:1367] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec: Time spent bootstrapping tablet: real 0.023s user 0.023s sys 0.001s
I20240502 20:47:04.604758 26997 raft_consensus.cc:348] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } }
I20240502 20:47:04.604879 26553 tablet_copy_service.cc:342] P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Request end of tablet copy session c8ef2de5cb854720b1090ff20c2fd13c-1409e44c98f44bd888eea4cee4f1163a received from {username='slave'} at 127.0.0.1:49944
I20240502 20:47:04.604923 26997 raft_consensus.cc:729] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2c2b4bb6fbe94bb8bea2eb59520f5fec, State: Initialized, Role: LEARNER
I20240502 20:47:04.604954 26553 tablet_copy_service.cc:433] P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: ending tablet copy session c8ef2de5cb854720b1090ff20c2fd13c-1409e44c98f44bd888eea4cee4f1163a on tablet 1409e44c98f44bd888eea4cee4f1163a with peer c8ef2de5cb854720b1090ff20c2fd13c
I20240502 20:47:04.605049 26997 consensus_queue.cc:260] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } }
I20240502 20:47:04.605748 26997 ts_tablet_manager.cc:1398] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:47:04.606186 26477 tablet_copy_service.cc:342] P ddb0f390a80f4c059c1a99a167c7d830: Request end of tablet copy session 2c2b4bb6fbe94bb8bea2eb59520f5fec-8ea4b2412a3a420f93d384aa48117c60 received from {username='slave'} at 127.0.0.1:56918
I20240502 20:47:04.606247 26477 tablet_copy_service.cc:433] P ddb0f390a80f4c059c1a99a167c7d830: ending tablet copy session 2c2b4bb6fbe94bb8bea2eb59520f5fec-8ea4b2412a3a420f93d384aa48117c60 on tablet 8ea4b2412a3a420f93d384aa48117c60 with peer 2c2b4bb6fbe94bb8bea2eb59520f5fec
I20240502 20:47:04.661891 27003 tablet_copy_client.cc:523] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:47:04.663472 27003 tablet_bootstrap.cc:492] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f: Bootstrap starting.
I20240502 20:47:04.687460 27003 tablet_bootstrap.cc:492] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=116 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:47:04.687762 27003 tablet_bootstrap.cc:492] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f: Bootstrap complete.
I20240502 20:47:04.687958 27003 ts_tablet_manager.cc:1367] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f: Time spent bootstrapping tablet: real 0.025s user 0.017s sys 0.007s
I20240502 20:47:04.688663 27003 raft_consensus.cc:348] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } }
I20240502 20:47:04.688809 27003 raft_consensus.cc:729] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ae30e3bd7dc1421998cf9258aaa6bf7f, State: Initialized, Role: LEARNER
I20240502 20:47:04.688933 27003 consensus_queue.cc:260] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } }
I20240502 20:47:04.689333 27003 ts_tablet_manager.cc:1398] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20240502 20:47:04.689819 26477 tablet_copy_service.cc:342] P ddb0f390a80f4c059c1a99a167c7d830: Request end of tablet copy session ae30e3bd7dc1421998cf9258aaa6bf7f-5db41d5853a34dc594bafb0f4c53cd7d received from {username='slave'} at 127.0.0.1:56924
I20240502 20:47:04.689888 26477 tablet_copy_service.cc:433] P ddb0f390a80f4c059c1a99a167c7d830: ending tablet copy session ae30e3bd7dc1421998cf9258aaa6bf7f-5db41d5853a34dc594bafb0f4c53cd7d on tablet 5db41d5853a34dc594bafb0f4c53cd7d with peer ae30e3bd7dc1421998cf9258aaa6bf7f
I20240502 20:47:04.699019 27009 tablet_copy_client.cc:523] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:47:04.700875 27009 tablet_bootstrap.cc:492] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c: Bootstrap starting.
I20240502 20:47:04.723690 27009 tablet_bootstrap.cc:492] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=114 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:47:04.724022 27009 tablet_bootstrap.cc:492] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c: Bootstrap complete.
I20240502 20:47:04.724221 27009 ts_tablet_manager.cc:1367] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c: Time spent bootstrapping tablet: real 0.023s user 0.019s sys 0.005s
I20240502 20:47:04.724728 27009 raft_consensus.cc:348] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } }
I20240502 20:47:04.724872 27009 raft_consensus.cc:729] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8ef2de5cb854720b1090ff20c2fd13c, State: Initialized, Role: LEARNER
I20240502 20:47:04.724985 27009 consensus_queue.cc:260] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } }
I20240502 20:47:04.725265 27009 ts_tablet_manager.cc:1398] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:47:04.725765 26553 tablet_copy_service.cc:342] P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Request end of tablet copy session c8ef2de5cb854720b1090ff20c2fd13c-1bc72d9e28894957a82f3b3ef37e6f08 received from {username='slave'} at 127.0.0.1:49944
I20240502 20:47:04.725844 26553 tablet_copy_service.cc:433] P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: ending tablet copy session c8ef2de5cb854720b1090ff20c2fd13c-1bc72d9e28894957a82f3b3ef37e6f08 on tablet 1bc72d9e28894957a82f3b3ef37e6f08 with peer c8ef2de5cb854720b1090ff20c2fd13c
I20240502 20:47:04.728623 27015 tablet_copy_client.cc:523] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:47:04.730170 27015 tablet_bootstrap.cc:492] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f: Bootstrap starting.
I20240502 20:47:04.752763 27015 tablet_bootstrap.cc:492] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=114 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:47:04.753036 27015 tablet_bootstrap.cc:492] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f: Bootstrap complete.
I20240502 20:47:04.753212 27015 ts_tablet_manager.cc:1367] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f: Time spent bootstrapping tablet: real 0.023s user 0.021s sys 0.001s
I20240502 20:47:04.753671 27021 tablet_copy_client.cc:523] T 7d1ff41afa6e46609e6a3eb277df18ad P 2c2b4bb6fbe94bb8bea2eb59520f5fec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:47:04.753741 27015 raft_consensus.cc:348] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } }
I20240502 20:47:04.753870 27015 raft_consensus.cc:729] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ae30e3bd7dc1421998cf9258aaa6bf7f, State: Initialized, Role: LEARNER
I20240502 20:47:04.753947 27015 consensus_queue.cc:260] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } }
I20240502 20:47:04.754171 27015 ts_tablet_manager.cc:1398] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:47:04.754591 26401 tablet_copy_service.cc:342] P 97406d5f0d1f4d59bdc784684baa1559: Request end of tablet copy session ae30e3bd7dc1421998cf9258aaa6bf7f-912fa9b5ccf74aadb519a8bf16659478 received from {username='slave'} at 127.0.0.1:52196
I20240502 20:47:04.754658 26401 tablet_copy_service.cc:433] P 97406d5f0d1f4d59bdc784684baa1559: ending tablet copy session ae30e3bd7dc1421998cf9258aaa6bf7f-912fa9b5ccf74aadb519a8bf16659478 on tablet 912fa9b5ccf74aadb519a8bf16659478 with peer ae30e3bd7dc1421998cf9258aaa6bf7f
I20240502 20:47:04.755199 27021 tablet_bootstrap.cc:492] T 7d1ff41afa6e46609e6a3eb277df18ad P 2c2b4bb6fbe94bb8bea2eb59520f5fec: Bootstrap starting.
I20240502 20:47:04.756805 26940 raft_consensus.cc:1203] T 1409e44c98f44bd888eea4cee4f1163a P c8ef2de5cb854720b1090ff20c2fd13c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29] Dedup: 1.29->[]
I20240502 20:47:04.777091 27021 tablet_bootstrap.cc:492] T 7d1ff41afa6e46609e6a3eb277df18ad P 2c2b4bb6fbe94bb8bea2eb59520f5fec: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=121 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:47:04.777377 27021 tablet_bootstrap.cc:492] T 7d1ff41afa6e46609e6a3eb277df18ad P 2c2b4bb6fbe94bb8bea2eb59520f5fec: Bootstrap complete.
I20240502 20:47:04.777594 27021 ts_tablet_manager.cc:1367] T 7d1ff41afa6e46609e6a3eb277df18ad P 2c2b4bb6fbe94bb8bea2eb59520f5fec: Time spent bootstrapping tablet: real 0.022s user 0.017s sys 0.003s
I20240502 20:47:04.778112 27021 raft_consensus.cc:348] T 7d1ff41afa6e46609e6a3eb277df18ad P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } }
I20240502 20:47:04.778252 27021 raft_consensus.cc:729] T 7d1ff41afa6e46609e6a3eb277df18ad P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2c2b4bb6fbe94bb8bea2eb59520f5fec, State: Initialized, Role: LEARNER
I20240502 20:47:04.778360 27021 consensus_queue.cc:260] T 7d1ff41afa6e46609e6a3eb277df18ad P 2c2b4bb6fbe94bb8bea2eb59520f5fec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } }
I20240502 20:47:04.778751 27021 ts_tablet_manager.cc:1398] T 7d1ff41afa6e46609e6a3eb277df18ad P 2c2b4bb6fbe94bb8bea2eb59520f5fec: Time spent starting tablet: real 0.001s user 0.003s sys 0.001s
I20240502 20:47:04.779139 26477 tablet_copy_service.cc:342] P ddb0f390a80f4c059c1a99a167c7d830: Request end of tablet copy session 2c2b4bb6fbe94bb8bea2eb59520f5fec-7d1ff41afa6e46609e6a3eb277df18ad received from {username='slave'} at 127.0.0.1:56918
I20240502 20:47:04.779206 26477 tablet_copy_service.cc:433] P ddb0f390a80f4c059c1a99a167c7d830: ending tablet copy session 2c2b4bb6fbe94bb8bea2eb59520f5fec-7d1ff41afa6e46609e6a3eb277df18ad on tablet 7d1ff41afa6e46609e6a3eb277df18ad with peer 2c2b4bb6fbe94bb8bea2eb59520f5fec
I20240502 20:47:04.818536 26860 raft_consensus.cc:1203] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20240502 20:47:04.917889 26787 raft_consensus.cc:1203] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240502 20:47:05.078465 26940 raft_consensus.cc:1203] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29] Dedup: 1.29->[]
I20240502 20:47:05.087666 26787 raft_consensus.cc:1203] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29] Dedup: 1.29->[]
I20240502 20:47:05.102030 26341 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:47:05.103298 26341 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 20:47:05.103432 26341 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:47:05.224388 27044 raft_consensus.cc:1050] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: attempting to promote NON_VOTER c8ef2de5cb854720b1090ff20c2fd13c to VOTER
I20240502 20:47:05.224795 27044 consensus_queue.cc:237] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } }
I20240502 20:47:05.226127 26461 raft_consensus.cc:1261] T 1409e44c98f44bd888eea4cee4f1163a P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 20:47:05.226259 26940 raft_consensus.cc:1261] T 1409e44c98f44bd888eea4cee4f1163a P c8ef2de5cb854720b1090ff20c2fd13c [term 1 LEARNER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 20:47:05.226279 26391 raft_consensus.cc:1261] T 1409e44c98f44bd888eea4cee4f1163a P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 20:47:05.226576 27046 consensus_queue.cc:1035] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 20:47:05.226989 27043 consensus_queue.cc:1035] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 20:47:05.227351 26980 consensus_queue.cc:1035] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 20:47:05.228749 26860 raft_consensus.cc:1203] T 7d1ff41afa6e46609e6a3eb277df18ad P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29] Dedup: 1.29->[]
I20240502 20:47:05.229372 26980 raft_consensus.cc:2932] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:05.229749 26391 raft_consensus.cc:2932] T 1409e44c98f44bd888eea4cee4f1163a P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:05.230597 26461 raft_consensus.cc:2932] T 1409e44c98f44bd888eea4cee4f1163a P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:05.231953 26285 catalog_manager.cc:5484] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 reported cstate change: config changed from index 29 to 30, c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:47:05.232566 26940 raft_consensus.cc:2932] T 1409e44c98f44bd888eea4cee4f1163a P c8ef2de5cb854720b1090ff20c2fd13c [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:05.236618 26541 consensus_queue.cc:237] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } }
I20240502 20:47:05.237586 26391 raft_consensus.cc:1261] T 1409e44c98f44bd888eea4cee4f1163a P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240502 20:47:05.237586 26940 raft_consensus.cc:1261] T 1409e44c98f44bd888eea4cee4f1163a P c8ef2de5cb854720b1090ff20c2fd13c [term 1 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240502 20:47:05.237926 27046 consensus_queue.cc:1035] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 20:47:05.238457 27046 consensus_queue.cc:1035] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 20:47:05.239374 27044 raft_consensus.cc:2932] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:05.239734 26390 raft_consensus.cc:2932] T 1409e44c98f44bd888eea4cee4f1163a P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:05.241022 26272 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1409e44c98f44bd888eea4cee4f1163a with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:47:05.241657 26283 catalog_manager.cc:5484] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 reported cstate change: config changed from index 30 to 31, VOTER ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130) evicted. New cstate: current_term: 1 leader_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:05.242221 26940 raft_consensus.cc:2932] T 1409e44c98f44bd888eea4cee4f1163a P c8ef2de5cb854720b1090ff20c2fd13c [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:05.244253 26449 tablet_service.cc:1514] Processing DeleteTablet for tablet 1409e44c98f44bd888eea4cee4f1163a with delete_type TABLET_DATA_TOMBSTONED (TS ddb0f390a80f4c059c1a99a167c7d830 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:56876
I20240502 20:47:05.244694 27054 tablet_replica.cc:330] T 1409e44c98f44bd888eea4cee4f1163a P ddb0f390a80f4c059c1a99a167c7d830: stopping tablet replica
I20240502 20:47:05.244843 27054 raft_consensus.cc:2229] T 1409e44c98f44bd888eea4cee4f1163a P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:05.245201 27054 raft_consensus.cc:2258] T 1409e44c98f44bd888eea4cee4f1163a P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:05.245915 27054 ts_tablet_manager.cc:1875] T 1409e44c98f44bd888eea4cee4f1163a P ddb0f390a80f4c059c1a99a167c7d830: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:05.247911 27054 ts_tablet_manager.cc:1888] T 1409e44c98f44bd888eea4cee4f1163a P ddb0f390a80f4c059c1a99a167c7d830: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.30
I20240502 20:47:05.247982 27054 log.cc:1196] T 1409e44c98f44bd888eea4cee4f1163a P ddb0f390a80f4c059c1a99a167c7d830: Deleting WAL directory at /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-1-root/wals/1409e44c98f44bd888eea4cee4f1163a
I20240502 20:47:05.248466 26271 catalog_manager.cc:4830] TS ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519): tablet 1409e44c98f44bd888eea4cee4f1163a (table test-workload [id=cc2caf6496c0475da19150577095963d]) successfully deleted
I20240502 20:47:05.305693 27030 raft_consensus.cc:1050] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559: attempting to promote NON_VOTER ae30e3bd7dc1421998cf9258aaa6bf7f to VOTER
I20240502 20:47:05.306186 27030 consensus_queue.cc:237] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } }
I20240502 20:47:05.307653 26541 raft_consensus.cc:1261] T 912fa9b5ccf74aadb519a8bf16659478 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 20:47:05.307752 26787 raft_consensus.cc:1261] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f [term 1 LEARNER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 20:47:05.308072 27058 consensus_queue.cc:1035] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 20:47:05.308383 26464 raft_consensus.cc:1261] T 912fa9b5ccf74aadb519a8bf16659478 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 20:47:05.308526 27030 consensus_queue.cc:1035] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 20:47:05.308701 27058 consensus_queue.cc:1035] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 20:47:05.310782 27058 raft_consensus.cc:2932] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } }
I20240502 20:47:05.311450 26464 raft_consensus.cc:2932] T 912fa9b5ccf74aadb519a8bf16659478 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } }
I20240502 20:47:05.312253 26541 raft_consensus.cc:2932] T 912fa9b5ccf74aadb519a8bf16659478 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } }
I20240502 20:47:05.313717 26787 raft_consensus.cc:2932] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } }
I20240502 20:47:05.314188 26281 catalog_manager.cc:5484] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 reported cstate change: config changed from index 27 to 28, ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "97406d5f0d1f4d59bdc784684baa1559" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:47:05.316768 26390 consensus_queue.cc:237] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } }
I20240502 20:47:05.318325 26541 raft_consensus.cc:1261] T 912fa9b5ccf74aadb519a8bf16659478 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 20:47:05.318607 26787 raft_consensus.cc:1261] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f [term 1 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 20:47:05.318718 27058 consensus_queue.cc:1035] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 20:47:05.318928 27055 consensus_queue.cc:1035] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 20:47:05.320322 27057 raft_consensus.cc:2932] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } }
I20240502 20:47:05.320614 26541 raft_consensus.cc:2932] T 912fa9b5ccf74aadb519a8bf16659478 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } }
I20240502 20:47:05.322259 26273 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 912fa9b5ccf74aadb519a8bf16659478 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:47:05.322613 26279 catalog_manager.cc:5484] T 912fa9b5ccf74aadb519a8bf16659478 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 reported cstate change: config changed from index 28 to 29, VOTER ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130) evicted. New cstate: current_term: 1 leader_uuid: "97406d5f0d1f4d59bdc784684baa1559" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } }
I20240502 20:47:05.323072 26787 raft_consensus.cc:2932] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } }
I20240502 20:47:05.324985 26449 tablet_service.cc:1514] Processing DeleteTablet for tablet 912fa9b5ccf74aadb519a8bf16659478 with delete_type TABLET_DATA_TOMBSTONED (TS ddb0f390a80f4c059c1a99a167c7d830 not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:56876
I20240502 20:47:05.325187 27054 tablet_replica.cc:330] T 912fa9b5ccf74aadb519a8bf16659478 P ddb0f390a80f4c059c1a99a167c7d830: stopping tablet replica
I20240502 20:47:05.325318 27054 raft_consensus.cc:2229] T 912fa9b5ccf74aadb519a8bf16659478 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:05.325503 27054 raft_consensus.cc:2258] T 912fa9b5ccf74aadb519a8bf16659478 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:05.326035 27054 ts_tablet_manager.cc:1875] T 912fa9b5ccf74aadb519a8bf16659478 P ddb0f390a80f4c059c1a99a167c7d830: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:05.328051 27054 ts_tablet_manager.cc:1888] T 912fa9b5ccf74aadb519a8bf16659478 P ddb0f390a80f4c059c1a99a167c7d830: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20240502 20:47:05.328114 27054 log.cc:1196] T 912fa9b5ccf74aadb519a8bf16659478 P ddb0f390a80f4c059c1a99a167c7d830: Deleting WAL directory at /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-1-root/wals/912fa9b5ccf74aadb519a8bf16659478
I20240502 20:47:05.328513 26271 catalog_manager.cc:4830] TS ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519): tablet 912fa9b5ccf74aadb519a8bf16659478 (table test-workload [id=cc2caf6496c0475da19150577095963d]) successfully deleted
I20240502 20:47:05.357664 27063 raft_consensus.cc:1050] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830: attempting to promote NON_VOTER 2c2b4bb6fbe94bb8bea2eb59520f5fec to VOTER
I20240502 20:47:05.358188 27063 consensus_queue.cc:237] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } }
I20240502 20:47:05.359517 26541 raft_consensus.cc:1261] T 8ea4b2412a3a420f93d384aa48117c60 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 20:47:05.359503 26860 raft_consensus.cc:1261] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 1 LEARNER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 20:47:05.359504 26390 raft_consensus.cc:1261] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 20:47:05.359911 27065 consensus_queue.cc:1035] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 20:47:05.360119 27038 consensus_queue.cc:1035] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 20:47:05.360423 27063 consensus_queue.cc:1035] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 20:47:05.361902 27063 raft_consensus.cc:2932] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:05.362390 26390 raft_consensus.cc:2932] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:05.364063 26860 raft_consensus.cc:2932] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:05.364187 26541 raft_consensus.cc:2932] T 8ea4b2412a3a420f93d384aa48117c60 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:05.365842 26285 catalog_manager.cc:5484] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559 reported cstate change: config changed from index 28 to 29, 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ddb0f390a80f4c059c1a99a167c7d830" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:05.369391 26461 consensus_queue.cc:237] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } }
I20240502 20:47:05.370546 26390 raft_consensus.cc:1261] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 20:47:05.370656 26860 raft_consensus.cc:1261] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 20:47:05.370963 27038 consensus_queue.cc:1035] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 20:47:05.371279 27038 consensus_queue.cc:1035] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 20:47:05.372762 27065 raft_consensus.cc:2932] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:05.373127 26390 raft_consensus.cc:2932] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:05.373174 26860 raft_consensus.cc:2932] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:05.374275 26271 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8ea4b2412a3a420f93d384aa48117c60 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:47:05.375254 26281 catalog_manager.cc:5484] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec reported cstate change: config changed from index 29 to 30, VOTER bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131) evicted. New cstate: current_term: 1 leader_uuid: "ddb0f390a80f4c059c1a99a167c7d830" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:05.377455 26526 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ea4b2412a3a420f93d384aa48117c60 with delete_type TABLET_DATA_TOMBSTONED (TS bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:49884
I20240502 20:47:05.377877 27069 tablet_replica.cc:330] T 8ea4b2412a3a420f93d384aa48117c60 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: stopping tablet replica
I20240502 20:47:05.378034 27069 raft_consensus.cc:2229] T 8ea4b2412a3a420f93d384aa48117c60 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:05.378222 27069 raft_consensus.cc:2258] T 8ea4b2412a3a420f93d384aa48117c60 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:05.378849 27069 ts_tablet_manager.cc:1875] T 8ea4b2412a3a420f93d384aa48117c60 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:05.381137 27069 ts_tablet_manager.cc:1888] T 8ea4b2412a3a420f93d384aa48117c60 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20240502 20:47:05.381222 27069 log.cc:1196] T 8ea4b2412a3a420f93d384aa48117c60 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Deleting WAL directory at /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-2-root/wals/8ea4b2412a3a420f93d384aa48117c60
I20240502 20:47:05.381781 26272 catalog_manager.cc:4830] TS bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881): tablet 8ea4b2412a3a420f93d384aa48117c60 (table test-workload [id=cc2caf6496c0475da19150577095963d]) successfully deleted
I20240502 20:47:05.545076 27043 raft_consensus.cc:1050] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: attempting to promote NON_VOTER c8ef2de5cb854720b1090ff20c2fd13c to VOTER
I20240502 20:47:05.545548 27043 consensus_queue.cc:237] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } }
I20240502 20:47:05.546913 26464 raft_consensus.cc:1261] T 1bc72d9e28894957a82f3b3ef37e6f08 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 20:47:05.546940 26390 raft_consensus.cc:1261] T 1bc72d9e28894957a82f3b3ef37e6f08 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 20:47:05.546913 26940 raft_consensus.cc:1261] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c [term 1 LEARNER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 20:47:05.547380 27044 consensus_queue.cc:1035] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 20:47:05.547578 27046 consensus_queue.cc:1035] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 20:47:05.547768 26980 consensus_queue.cc:1035] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 20:47:05.549912 26980 raft_consensus.cc:2932] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:05.550230 26464 raft_consensus.cc:2932] T 1bc72d9e28894957a82f3b3ef37e6f08 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:05.550469 26390 raft_consensus.cc:2932] T 1bc72d9e28894957a82f3b3ef37e6f08 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:05.552309 26279 catalog_manager.cc:5484] T 1bc72d9e28894957a82f3b3ef37e6f08 P ddb0f390a80f4c059c1a99a167c7d830 reported cstate change: config changed from index 29 to 30, c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:05.552789 26940 raft_consensus.cc:2932] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:05.557089 26541 consensus_queue.cc:237] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } }
I20240502 20:47:05.558208 26940 raft_consensus.cc:1261] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c [term 1 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240502 20:47:05.558208 26464 raft_consensus.cc:1261] T 1bc72d9e28894957a82f3b3ef37e6f08 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240502 20:47:05.558722 27043 consensus_queue.cc:1035] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 20:47:05.558892 26980 consensus_queue.cc:1035] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 20:47:05.560532 27046 raft_consensus.cc:2932] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:05.560871 26940 raft_consensus.cc:2932] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:05.561996 26272 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1bc72d9e28894957a82f3b3ef37e6f08 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:47:05.563038 26283 catalog_manager.cc:5484] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c reported cstate change: config changed from index 30 to 31, VOTER 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129) evicted. New cstate: current_term: 1 leader_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:05.563414 26464 raft_consensus.cc:2932] T 1bc72d9e28894957a82f3b3ef37e6f08 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:05.565261 27038 raft_consensus.cc:1050] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830: attempting to promote NON_VOTER ae30e3bd7dc1421998cf9258aaa6bf7f to VOTER
I20240502 20:47:05.565650 27038 consensus_queue.cc:237] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } }
I20240502 20:47:05.566108 26373 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bc72d9e28894957a82f3b3ef37e6f08 with delete_type TABLET_DATA_TOMBSTONED (TS 97406d5f0d1f4d59bdc784684baa1559 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:52152
I20240502 20:47:05.566923 27075 tablet_replica.cc:330] T 1bc72d9e28894957a82f3b3ef37e6f08 P 97406d5f0d1f4d59bdc784684baa1559: stopping tablet replica
I20240502 20:47:05.567078 26390 raft_consensus.cc:1261] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 20:47:05.567114 26787 raft_consensus.cc:1261] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f [term 1 LEARNER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 20:47:05.567245 27075 raft_consensus.cc:2229] T 1bc72d9e28894957a82f3b3ef37e6f08 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:05.567445 27075 raft_consensus.cc:2258] T 1bc72d9e28894957a82f3b3ef37e6f08 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:05.567406 27063 consensus_queue.cc:1035] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 20:47:05.567508 26538 raft_consensus.cc:1261] T 5db41d5853a34dc594bafb0f4c53cd7d P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 20:47:05.567703 27062 consensus_queue.cc:1035] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 20:47:05.567901 27063 consensus_queue.cc:1035] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 20:47:05.568265 27075 ts_tablet_manager.cc:1875] T 1bc72d9e28894957a82f3b3ef37e6f08 P 97406d5f0d1f4d59bdc784684baa1559: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:05.570055 27062 raft_consensus.cc:2932] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } }
I20240502 20:47:05.570644 27075 ts_tablet_manager.cc:1888] T 1bc72d9e28894957a82f3b3ef37e6f08 P 97406d5f0d1f4d59bdc784684baa1559: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.30
I20240502 20:47:05.570744 27075 log.cc:1196] T 1bc72d9e28894957a82f3b3ef37e6f08 P 97406d5f0d1f4d59bdc784684baa1559: Deleting WAL directory at /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-0-root/wals/1bc72d9e28894957a82f3b3ef37e6f08
I20240502 20:47:05.571235 26273 catalog_manager.cc:4830] TS 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989): tablet 1bc72d9e28894957a82f3b3ef37e6f08 (table test-workload [id=cc2caf6496c0475da19150577095963d]) successfully deleted
I20240502 20:47:05.571841 26538 raft_consensus.cc:2932] T 5db41d5853a34dc594bafb0f4c53cd7d P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } }
I20240502 20:47:05.572026 26787 raft_consensus.cc:2932] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } }
I20240502 20:47:05.573498 26390 raft_consensus.cc:2932] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } }
I20240502 20:47:05.573812 26285 catalog_manager.cc:5484] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 reported cstate change: config changed from index 29 to 30, ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ddb0f390a80f4c059c1a99a167c7d830" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:47:05.578403 26461 consensus_queue.cc:237] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } }
I20240502 20:47:05.579486 26787 raft_consensus.cc:1261] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240502 20:47:05.579555 26390 raft_consensus.cc:1261] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240502 20:47:05.579862 27062 consensus_queue.cc:1035] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 20:47:05.580060 27038 consensus_queue.cc:1035] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 20:47:05.581408 27065 raft_consensus.cc:2932] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } }
I20240502 20:47:05.581780 26787 raft_consensus.cc:2932] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } }
I20240502 20:47:05.582831 26271 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5db41d5853a34dc594bafb0f4c53cd7d with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:47:05.583426 26390 raft_consensus.cc:2932] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } }
I20240502 20:47:05.583642 26279 catalog_manager.cc:5484] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 reported cstate change: config changed from index 30 to 31, VOTER bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131) evicted. New cstate: current_term: 1 leader_uuid: "ddb0f390a80f4c059c1a99a167c7d830" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:47:05.585870 26526 tablet_service.cc:1514] Processing DeleteTablet for tablet 5db41d5853a34dc594bafb0f4c53cd7d with delete_type TABLET_DATA_TOMBSTONED (TS bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:49884
I20240502 20:47:05.586086 27069 tablet_replica.cc:330] T 5db41d5853a34dc594bafb0f4c53cd7d P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: stopping tablet replica
I20240502 20:47:05.586234 27069 raft_consensus.cc:2229] T 5db41d5853a34dc594bafb0f4c53cd7d P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:05.586365 27069 raft_consensus.cc:2258] T 5db41d5853a34dc594bafb0f4c53cd7d P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:05.587076 27069 ts_tablet_manager.cc:1875] T 5db41d5853a34dc594bafb0f4c53cd7d P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:05.589241 27069 ts_tablet_manager.cc:1888] T 5db41d5853a34dc594bafb0f4c53cd7d P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.30
I20240502 20:47:05.589331 27069 log.cc:1196] T 5db41d5853a34dc594bafb0f4c53cd7d P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Deleting WAL directory at /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-2-root/wals/5db41d5853a34dc594bafb0f4c53cd7d
I20240502 20:47:05.589867 26272 catalog_manager.cc:4830] TS bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881): tablet 5db41d5853a34dc594bafb0f4c53cd7d (table test-workload [id=cc2caf6496c0475da19150577095963d]) successfully deleted
I20240502 20:47:05.711074 27038 raft_consensus.cc:1050] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830: attempting to promote NON_VOTER 2c2b4bb6fbe94bb8bea2eb59520f5fec to VOTER
I20240502 20:47:05.711472 27038 consensus_queue.cc:237] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } }
I20240502 20:47:05.712826 26390 raft_consensus.cc:1261] T 7d1ff41afa6e46609e6a3eb277df18ad P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 20:47:05.712833 26538 raft_consensus.cc:1261] T 7d1ff41afa6e46609e6a3eb277df18ad P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 20:47:05.713191 27065 consensus_queue.cc:1035] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 20:47:05.713670 27065 consensus_queue.cc:1035] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 20:47:05.713928 26860 raft_consensus.cc:1261] T 7d1ff41afa6e46609e6a3eb277df18ad P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 1 LEARNER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 20:47:05.714372 27065 consensus_queue.cc:1035] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 20:47:05.715646 27062 raft_consensus.cc:2932] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:05.716012 26538 raft_consensus.cc:2932] T 7d1ff41afa6e46609e6a3eb277df18ad P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:05.718025 26390 raft_consensus.cc:2932] T 7d1ff41afa6e46609e6a3eb277df18ad P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:05.718058 26281 catalog_manager.cc:5484] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 reported cstate change: config changed from index 29 to 30, 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ddb0f390a80f4c059c1a99a167c7d830" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:05.718538 26860 raft_consensus.cc:2932] T 7d1ff41afa6e46609e6a3eb277df18ad P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:05.722488 26461 consensus_queue.cc:237] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } }
I20240502 20:47:05.723457 26538 raft_consensus.cc:1261] T 7d1ff41afa6e46609e6a3eb277df18ad P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240502 20:47:05.723457 26860 raft_consensus.cc:1261] T 7d1ff41afa6e46609e6a3eb277df18ad P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240502 20:47:05.723819 27065 consensus_queue.cc:1035] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 20:47:05.723979 27062 consensus_queue.cc:1035] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 20:47:05.725317 27063 raft_consensus.cc:2932] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:05.725685 26538 raft_consensus.cc:2932] T 7d1ff41afa6e46609e6a3eb277df18ad P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:05.727219 26860 raft_consensus.cc:2932] T 7d1ff41afa6e46609e6a3eb277df18ad P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:05.727443 26285 catalog_manager.cc:5484] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 reported cstate change: config changed from index 30 to 31, VOTER 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129) evicted. New cstate: current_term: 1 leader_uuid: "ddb0f390a80f4c059c1a99a167c7d830" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:47:05.728384 26271 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7d1ff41afa6e46609e6a3eb277df18ad with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:47:05.729609 26373 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d1ff41afa6e46609e6a3eb277df18ad with delete_type TABLET_DATA_TOMBSTONED (TS 97406d5f0d1f4d59bdc784684baa1559 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:52152
I20240502 20:47:05.729832 27075 tablet_replica.cc:330] T 7d1ff41afa6e46609e6a3eb277df18ad P 97406d5f0d1f4d59bdc784684baa1559: stopping tablet replica
I20240502 20:47:05.729977 27075 raft_consensus.cc:2229] T 7d1ff41afa6e46609e6a3eb277df18ad P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:05.730111 27075 raft_consensus.cc:2258] T 7d1ff41afa6e46609e6a3eb277df18ad P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:05.730775 27075 ts_tablet_manager.cc:1875] T 7d1ff41afa6e46609e6a3eb277df18ad P 97406d5f0d1f4d59bdc784684baa1559: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:05.732481 27075 ts_tablet_manager.cc:1888] T 7d1ff41afa6e46609e6a3eb277df18ad P 97406d5f0d1f4d59bdc784684baa1559: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.30
I20240502 20:47:05.732533 27075 log.cc:1196] T 7d1ff41afa6e46609e6a3eb277df18ad P 97406d5f0d1f4d59bdc784684baa1559: Deleting WAL directory at /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-0-root/wals/7d1ff41afa6e46609e6a3eb277df18ad
I20240502 20:47:05.732949 26273 catalog_manager.cc:4830] TS 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989): tablet 7d1ff41afa6e46609e6a3eb277df18ad (table test-workload [id=cc2caf6496c0475da19150577095963d]) successfully deleted
I20240502 20:47:06.734256 26538 consensus_queue.cc:237] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } }
I20240502 20:47:06.735731 26461 raft_consensus.cc:1261] T 082557bf92e4443cbb9d0c50a89d24c4 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 20:47:06.735823 26390 raft_consensus.cc:1261] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
W20240502 20:47:06.735864 26507 consensus_peers.cc:485] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 -> Peer c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134:45157): Couldn't send request to peer c8ef2de5cb854720b1090ff20c2fd13c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 082557bf92e4443cbb9d0c50a89d24c4. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:06.736218 26980 consensus_queue.cc:1035] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 20:47:06.736410 27084 consensus_queue.cc:1035] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 20:47:06.738252 27084 raft_consensus.cc:2932] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } } }
I20240502 20:47:06.738452 26390 raft_consensus.cc:2932] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } } }
I20240502 20:47:06.738570 26461 raft_consensus.cc:2932] T 082557bf92e4443cbb9d0c50a89d24c4 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } } }
I20240502 20:47:06.740689 26279 catalog_manager.cc:5484] T 082557bf92e4443cbb9d0c50a89d24c4 P ddb0f390a80f4c059c1a99a167c7d830 reported cstate change: config changed from index -1 to 29, NON_VOTER c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) added. New cstate: current_term: 1 leader_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } } }
I20240502 20:47:06.741307 26461 consensus_queue.cc:237] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } }
I20240502 20:47:06.742856 26390 raft_consensus.cc:1261] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 20:47:06.742974 26538 raft_consensus.cc:1261] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 20:47:06.743382 27086 consensus_queue.cc:1035] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 20:47:06.743712 27038 consensus_queue.cc:1035] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
W20240502 20:47:06.744109 26430 consensus_peers.cc:485] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 -> Peer 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133:43257): Couldn't send request to peer 2c2b4bb6fbe94bb8bea2eb59520f5fec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32631c89419245dea46abe6baa5dc295. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:06.745995 27092 raft_consensus.cc:2932] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } } }
I20240502 20:47:06.746392 26390 raft_consensus.cc:2932] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } } }
I20240502 20:47:06.746870 26538 raft_consensus.cc:2932] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } } }
I20240502 20:47:06.748772 26283 catalog_manager.cc:5484] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 reported cstate change: config changed from index -1 to 29, NON_VOTER 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) added. New cstate: current_term: 1 leader_uuid: "ddb0f390a80f4c059c1a99a167c7d830" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:06.748868 26538 consensus_queue.cc:237] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } }
I20240502 20:47:06.750957 26390 raft_consensus.cc:1261] T ca0df961aea849f4bc2ea340349f8848 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 20:47:06.751117 26461 raft_consensus.cc:1261] T ca0df961aea849f4bc2ea340349f8848 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 20:47:06.751459 27084 consensus_queue.cc:1035] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 20:47:06.751686 26980 consensus_queue.cc:1035] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 20:47:06.753553 27084 raft_consensus.cc:2932] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } } }
I20240502 20:47:06.753964 26390 raft_consensus.cc:2932] T ca0df961aea849f4bc2ea340349f8848 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } } }
I20240502 20:47:06.755700 26461 raft_consensus.cc:2932] T ca0df961aea849f4bc2ea340349f8848 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } } }
I20240502 20:47:06.756590 26285 catalog_manager.cc:5484] T ca0df961aea849f4bc2ea340349f8848 P 97406d5f0d1f4d59bdc784684baa1559 reported cstate change: config changed from index -1 to 27, NON_VOTER ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) added. New cstate: current_term: 1 leader_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } } }
W20240502 20:47:06.757261 26508 consensus_peers.cc:485] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 -> Peer ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132:37635): Couldn't send request to peer ae30e3bd7dc1421998cf9258aaa6bf7f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ca0df961aea849f4bc2ea340349f8848. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:06.757800 26461 consensus_queue.cc:237] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } }
I20240502 20:47:06.759117 26390 raft_consensus.cc:1261] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240502 20:47:06.759128 26860 raft_consensus.cc:1261] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
W20240502 20:47:06.759384 26433 consensus_peers.cc:485] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 -> Peer ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132:37635): Couldn't send request to peer ae30e3bd7dc1421998cf9258aaa6bf7f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ea4b2412a3a420f93d384aa48117c60. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:06.759497 27038 consensus_queue.cc:1035] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 20:47:06.759676 27092 consensus_queue.cc:1035] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 20:47:06.761301 27092 raft_consensus.cc:2932] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, NON_VOTER ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } } }
I20240502 20:47:06.761709 26390 raft_consensus.cc:2932] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, NON_VOTER ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } } }
I20240502 20:47:06.761859 26860 raft_consensus.cc:2932] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, NON_VOTER ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } } }
I20240502 20:47:06.763746 26279 catalog_manager.cc:5484] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 reported cstate change: config changed from index 30 to 31, NON_VOTER ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) added. New cstate: current_term: 1 leader_uuid: "ddb0f390a80f4c059c1a99a167c7d830" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:06.764210 26461 consensus_queue.cc:237] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } }
I20240502 20:47:06.765797 26860 raft_consensus.cc:1261] T 7d1ff41afa6e46609e6a3eb277df18ad P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20240502 20:47:06.765937 26538 raft_consensus.cc:1261] T 7d1ff41afa6e46609e6a3eb277df18ad P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20240502 20:47:06.766235 27092 consensus_queue.cc:1035] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 20:47:06.766475 27086 consensus_queue.cc:1035] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 20:47:06.768071 27086 raft_consensus.cc:2932] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [term 1 LEADER]: Committing config change with OpId 1.32: config changed from index 31 to 32, NON_VOTER c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } } }
I20240502 20:47:06.768569 26538 raft_consensus.cc:2932] T 7d1ff41afa6e46609e6a3eb277df18ad P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, NON_VOTER c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } } }
I20240502 20:47:06.769692 26860 raft_consensus.cc:2932] T 7d1ff41afa6e46609e6a3eb277df18ad P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, NON_VOTER c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } } }
W20240502 20:47:06.771108 26432 consensus_peers.cc:485] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 -> Peer c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134:45157): Couldn't send request to peer c8ef2de5cb854720b1090ff20c2fd13c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d1ff41afa6e46609e6a3eb277df18ad. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:06.771139 26390 consensus_queue.cc:237] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } }
I20240502 20:47:06.771363 26283 catalog_manager.cc:5484] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 reported cstate change: config changed from index 31 to 32, NON_VOTER c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) added. New cstate: current_term: 1 leader_uuid: "ddb0f390a80f4c059c1a99a167c7d830" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:06.772989 26538 raft_consensus.cc:1261] T 7d7f28e11e1f42959251a7b03011f5dc P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 20:47:06.773484 26461 raft_consensus.cc:1261] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 20:47:06.773650 27113 consensus_queue.cc:1035] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 20:47:06.774132 27055 consensus_queue.cc:1035] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 20:47:06.776283 27055 raft_consensus.cc:2932] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } } }
I20240502 20:47:06.777423 26538 raft_consensus.cc:2932] T 7d7f28e11e1f42959251a7b03011f5dc P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } } }
W20240502 20:47:06.777905 26353 consensus_peers.cc:485] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 -> Peer 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133:43257): Couldn't send request to peer 2c2b4bb6fbe94bb8bea2eb59520f5fec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d7f28e11e1f42959251a7b03011f5dc. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:06.778446 26461 raft_consensus.cc:2932] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } } }
I20240502 20:47:06.778643 26285 catalog_manager.cc:5484] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 reported cstate change: config changed from index -1 to 29, NON_VOTER 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) added. New cstate: current_term: 1 leader_uuid: "97406d5f0d1f4d59bdc784684baa1559" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:07.103650 26341 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:47:07.114771 26541 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "082557bf92e4443cbb9d0c50a89d24c4"
dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3"
mode: GRACEFUL
new_leader_uuid: "97406d5f0d1f4d59bdc784684baa1559"
from {username='slave'} at 127.0.0.1:35504
I20240502 20:47:07.114970 26541 raft_consensus.cc:595] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 LEADER]: Received request to transfer leadership to TS 97406d5f0d1f4d59bdc784684baa1559
I20240502 20:47:07.115731 26538 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1409e44c98f44bd888eea4cee4f1163a"
dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3"
mode: GRACEFUL
new_leader_uuid: "c8ef2de5cb854720b1090ff20c2fd13c"
from {username='slave'} at 127.0.0.1:35504
I20240502 20:47:07.115854 26538 raft_consensus.cc:595] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 LEADER]: Received request to transfer leadership to TS c8ef2de5cb854720b1090ff20c2fd13c
I20240502 20:47:07.119961 26390 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "23f60af679e74f078fa50af755a9fbf6"
dest_uuid: "97406d5f0d1f4d59bdc784684baa1559"
mode: GRACEFUL
new_leader_uuid: "ddb0f390a80f4c059c1a99a167c7d830"
from {username='slave'} at 127.0.0.1:45110
I20240502 20:47:07.120101 26390 raft_consensus.cc:595] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 LEADER]: Received request to transfer leadership to TS ddb0f390a80f4c059c1a99a167c7d830
I20240502 20:47:07.124153 26461 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "32631c89419245dea46abe6baa5dc295"
dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830"
mode: GRACEFUL
new_leader_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3"
from {username='slave'} at 127.0.0.1:43758
I20240502 20:47:07.124286 26461 raft_consensus.cc:595] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 LEADER]: Received request to transfer leadership to TS bdcd7bbd5fa14f21bbf0a7a3cdc33cb3
I20240502 20:47:07.124656 27092 raft_consensus.cc:979] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830: : Instructing follower bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 to start an election
I20240502 20:47:07.124768 27092 raft_consensus.cc:1067] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 LEADER]: Signalling peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 to start an election
I20240502 20:47:07.125129 26461 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8ea4b2412a3a420f93d384aa48117c60"
dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830"
mode: GRACEFUL
new_leader_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec"
from {username='slave'} at 127.0.0.1:43758
I20240502 20:47:07.125239 26461 raft_consensus.cc:595] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 LEADER]: Received request to transfer leadership to TS 2c2b4bb6fbe94bb8bea2eb59520f5fec
I20240502 20:47:07.125250 26541 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "32631c89419245dea46abe6baa5dc295"
dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3"
from {username='slave'} at 127.0.0.1:49912
I20240502 20:47:07.125381 26541 raft_consensus.cc:482] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:47:07.125429 26541 raft_consensus.cc:3037] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:07.125870 26390 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "912fa9b5ccf74aadb519a8bf16659478"
dest_uuid: "97406d5f0d1f4d59bdc784684baa1559"
mode: GRACEFUL
new_leader_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f"
from {username='slave'} at 127.0.0.1:45110
I20240502 20:47:07.125964 26390 raft_consensus.cc:595] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 LEADER]: Received request to transfer leadership to TS ae30e3bd7dc1421998cf9258aaa6bf7f
I20240502 20:47:07.126231 26341 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 6
I20240502 20:47:07.126372 26341 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:47:07.126492 26541 raft_consensus.cc:504] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } }
I20240502 20:47:07.126843 26541 leader_election.cc:288] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [CANDIDATE]: Term 2 election: Requested vote from peers ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519), 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989)
I20240502 20:47:07.127091 26461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32631c89419245dea46abe6baa5dc295" candidate_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830"
I20240502 20:47:07.127177 26461 raft_consensus.cc:3032] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:47:07.127215 26461 raft_consensus.cc:729] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ddb0f390a80f4c059c1a99a167c7d830, State: Running, Role: LEADER
I20240502 20:47:07.127367 26461 consensus_queue.cc:260] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } }
I20240502 20:47:07.127518 26390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32631c89419245dea46abe6baa5dc295" candidate_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "97406d5f0d1f4d59bdc784684baa1559"
I20240502 20:47:07.127609 26461 raft_consensus.cc:3037] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:07.127628 26390 raft_consensus.cc:3037] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:07.128389 26390 raft_consensus.cc:2454] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 in term 2.
I20240502 20:47:07.128398 26461 raft_consensus.cc:2454] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 in term 2.
I20240502 20:47:07.128711 26505 leader_election.cc:302] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, ddb0f390a80f4c059c1a99a167c7d830; no voters:
I20240502 20:47:07.128886 27084 raft_consensus.cc:2781] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:47:07.128968 27084 raft_consensus.cc:686] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 LEADER]: Becoming Leader. State: Replica: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, State: Running, Role: LEADER
I20240502 20:47:07.129089 27084 consensus_queue.cc:237] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } }
I20240502 20:47:07.130832 26283 catalog_manager.cc:5484] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 reported cstate change: term changed from 1 to 2, leader changed from ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130) to bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131). New cstate: current_term: 2 leader_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:07.134498 26538 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "32631c89419245dea46abe6baa5dc295"
dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49884
I20240502 20:47:07.134616 26538 raft_consensus.cc:595] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 LEADER]: Received request to transfer leadership
I20240502 20:47:07.139700 26538 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "32631c89419245dea46abe6baa5dc295"
dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49884
I20240502 20:47:07.139820 26538 raft_consensus.cc:595] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 LEADER]: Received request to transfer leadership
W20240502 20:47:07.140074 26272 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 32631c89419245dea46abe6baa5dc295 already in progress [suppressed 2 similar messages]
I20240502 20:47:07.140884 26538 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "32631c89419245dea46abe6baa5dc295"
dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49884
I20240502 20:47:07.140996 26538 raft_consensus.cc:595] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 LEADER]: Received request to transfer leadership
I20240502 20:47:07.141088 27055 raft_consensus.cc:979] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559: : Instructing follower ddb0f390a80f4c059c1a99a167c7d830 to start an election
I20240502 20:47:07.141167 27055 raft_consensus.cc:1067] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 LEADER]: Signalling peer ddb0f390a80f4c059c1a99a167c7d830 to start an election
I20240502 20:47:07.141536 26464 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "23f60af679e74f078fa50af755a9fbf6"
dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830"
from {username='slave'} at 127.0.0.1:56890
I20240502 20:47:07.141667 26464 raft_consensus.cc:482] T 23f60af679e74f078fa50af755a9fbf6 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:47:07.141732 26464 raft_consensus.cc:3037] T 23f60af679e74f078fa50af755a9fbf6 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Advancing to term 2
W20240502 20:47:07.142727 26335 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 32631c89419245dea46abe6baa5dc295 already in progress
W20240502 20:47:07.142791 26335 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 32631c89419245dea46abe6baa5dc295 already in progress
I20240502 20:47:07.143893 26464 raft_consensus.cc:504] T 23f60af679e74f078fa50af755a9fbf6 P ddb0f390a80f4c059c1a99a167c7d830 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:07.144276 26464 leader_election.cc:288] T 23f60af679e74f078fa50af755a9fbf6 P ddb0f390a80f4c059c1a99a167c7d830 [CANDIDATE]: Term 2 election: Requested vote from peers 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989), bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881)
I20240502 20:47:07.144480 26390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f60af679e74f078fa50af755a9fbf6" candidate_uuid: "ddb0f390a80f4c059c1a99a167c7d830" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "97406d5f0d1f4d59bdc784684baa1559"
I20240502 20:47:07.144601 26390 raft_consensus.cc:3032] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:47:07.144646 26390 raft_consensus.cc:729] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 97406d5f0d1f4d59bdc784684baa1559, State: Running, Role: LEADER
I20240502 20:47:07.144681 26538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f60af679e74f078fa50af755a9fbf6" candidate_uuid: "ddb0f390a80f4c059c1a99a167c7d830" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3"
I20240502 20:47:07.144783 26538 raft_consensus.cc:3037] T 23f60af679e74f078fa50af755a9fbf6 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:07.144784 26390 consensus_queue.cc:260] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:07.144943 26390 raft_consensus.cc:3037] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:07.145540 26538 raft_consensus.cc:2454] T 23f60af679e74f078fa50af755a9fbf6 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddb0f390a80f4c059c1a99a167c7d830 in term 2.
I20240502 20:47:07.145689 26390 raft_consensus.cc:2454] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddb0f390a80f4c059c1a99a167c7d830 in term 2.
I20240502 20:47:07.145812 26431 leader_election.cc:302] T 23f60af679e74f078fa50af755a9fbf6 P ddb0f390a80f4c059c1a99a167c7d830 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, ddb0f390a80f4c059c1a99a167c7d830; no voters:
I20240502 20:47:07.145962 27092 raft_consensus.cc:2781] T 23f60af679e74f078fa50af755a9fbf6 P ddb0f390a80f4c059c1a99a167c7d830 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:47:07.146080 27092 raft_consensus.cc:686] T 23f60af679e74f078fa50af755a9fbf6 P ddb0f390a80f4c059c1a99a167c7d830 [term 2 LEADER]: Becoming Leader. State: Replica: ddb0f390a80f4c059c1a99a167c7d830, State: Running, Role: LEADER
I20240502 20:47:07.146219 27092 consensus_queue.cc:237] T 23f60af679e74f078fa50af755a9fbf6 P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:07.147709 26283 catalog_manager.cc:5484] T 23f60af679e74f078fa50af755a9fbf6 P ddb0f390a80f4c059c1a99a167c7d830 reported cstate change: term changed from 1 to 2, leader changed from 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129) to ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130). New cstate: current_term: 2 leader_uuid: "ddb0f390a80f4c059c1a99a167c7d830" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:07.220019 26980 raft_consensus.cc:979] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: : Instructing follower c8ef2de5cb854720b1090ff20c2fd13c to start an election
I20240502 20:47:07.220152 26980 raft_consensus.cc:1067] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 LEADER]: Signalling peer c8ef2de5cb854720b1090ff20c2fd13c to start an election
I20240502 20:47:07.220744 26939 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1409e44c98f44bd888eea4cee4f1163a"
dest_uuid: "c8ef2de5cb854720b1090ff20c2fd13c"
from {username='slave'} at 127.0.0.1:44676
I20240502 20:47:07.220888 26939 raft_consensus.cc:482] T 1409e44c98f44bd888eea4cee4f1163a P c8ef2de5cb854720b1090ff20c2fd13c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:47:07.220957 26939 raft_consensus.cc:3037] T 1409e44c98f44bd888eea4cee4f1163a P c8ef2de5cb854720b1090ff20c2fd13c [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:07.222129 26939 raft_consensus.cc:504] T 1409e44c98f44bd888eea4cee4f1163a P c8ef2de5cb854720b1090ff20c2fd13c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } }
I20240502 20:47:07.222604 26939 leader_election.cc:288] T 1409e44c98f44bd888eea4cee4f1163a P c8ef2de5cb854720b1090ff20c2fd13c [CANDIDATE]: Term 2 election: Requested vote from peers 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989), bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881)
I20240502 20:47:07.222918 27124 ts_tablet_manager.cc:902] T 7d1ff41afa6e46609e6a3eb277df18ad P c8ef2de5cb854720b1090ff20c2fd13c: Initiating tablet copy from peer ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519)
I20240502 20:47:07.222918 26538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1409e44c98f44bd888eea4cee4f1163a" candidate_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3"
I20240502 20:47:07.223039 26538 raft_consensus.cc:3032] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:47:07.223070 26538 raft_consensus.cc:729] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, State: Running, Role: LEADER
I20240502 20:47:07.223204 26538 consensus_queue.cc:260] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } }
I20240502 20:47:07.223378 26538 raft_consensus.cc:3037] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:07.223482 27124 tablet_copy_client.cc:308] T 7d1ff41afa6e46609e6a3eb277df18ad P c8ef2de5cb854720b1090ff20c2fd13c: tablet copy: Beginning tablet copy session from remote peer at address 127.22.104.130:35519
I20240502 20:47:07.224294 26538 raft_consensus.cc:2454] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8ef2de5cb854720b1090ff20c2fd13c in term 2.
I20240502 20:47:07.225428 27132 ts_tablet_manager.cc:902] T ca0df961aea849f4bc2ea340349f8848 P ae30e3bd7dc1421998cf9258aaa6bf7f: Initiating tablet copy from peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881)
I20240502 20:47:07.225618 26903 leader_election.cc:302] T 1409e44c98f44bd888eea4cee4f1163a P c8ef2de5cb854720b1090ff20c2fd13c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, c8ef2de5cb854720b1090ff20c2fd13c; no voters:
I20240502 20:47:07.225937 27134 raft_consensus.cc:2781] T 1409e44c98f44bd888eea4cee4f1163a P c8ef2de5cb854720b1090ff20c2fd13c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:47:07.226519 27131 ts_tablet_manager.cc:902] T 8ea4b2412a3a420f93d384aa48117c60 P ae30e3bd7dc1421998cf9258aaa6bf7f: Initiating tablet copy from peer ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519)
I20240502 20:47:07.227365 27131 tablet_copy_client.cc:308] T 8ea4b2412a3a420f93d384aa48117c60 P ae30e3bd7dc1421998cf9258aaa6bf7f: tablet copy: Beginning tablet copy session from remote peer at address 127.22.104.130:35519
I20240502 20:47:07.227497 27132 tablet_copy_client.cc:308] T ca0df961aea849f4bc2ea340349f8848 P ae30e3bd7dc1421998cf9258aaa6bf7f: tablet copy: Beginning tablet copy session from remote peer at address 127.22.104.131:38881
I20240502 20:47:07.228101 26477 tablet_copy_service.cc:140] P ddb0f390a80f4c059c1a99a167c7d830: Received BeginTabletCopySession request for tablet 7d1ff41afa6e46609e6a3eb277df18ad from peer c8ef2de5cb854720b1090ff20c2fd13c ({username='slave'} at 127.0.0.1:43774)
I20240502 20:47:07.228195 26477 tablet_copy_service.cc:161] P ddb0f390a80f4c059c1a99a167c7d830: Beginning new tablet copy session on tablet 7d1ff41afa6e46609e6a3eb277df18ad from peer c8ef2de5cb854720b1090ff20c2fd13c at {username='slave'} at 127.0.0.1:43774: session id = c8ef2de5cb854720b1090ff20c2fd13c-7d1ff41afa6e46609e6a3eb277df18ad
I20240502 20:47:07.228641 26476 tablet_copy_service.cc:140] P ddb0f390a80f4c059c1a99a167c7d830: Received BeginTabletCopySession request for tablet 8ea4b2412a3a420f93d384aa48117c60 from peer ae30e3bd7dc1421998cf9258aaa6bf7f ({username='slave'} at 127.0.0.1:56924)
I20240502 20:47:07.228724 26476 tablet_copy_service.cc:161] P ddb0f390a80f4c059c1a99a167c7d830: Beginning new tablet copy session on tablet 8ea4b2412a3a420f93d384aa48117c60 from peer ae30e3bd7dc1421998cf9258aaa6bf7f at {username='slave'} at 127.0.0.1:56924: session id = ae30e3bd7dc1421998cf9258aaa6bf7f-8ea4b2412a3a420f93d384aa48117c60
I20240502 20:47:07.229228 26477 tablet_copy_source_session.cc:205] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:47:07.229712 26476 tablet_copy_source_session.cc:205] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:47:07.230069 27124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d1ff41afa6e46609e6a3eb277df18ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:07.230557 27131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea4b2412a3a420f93d384aa48117c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:07.231480 26553 tablet_copy_service.cc:140] P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Received BeginTabletCopySession request for tablet ca0df961aea849f4bc2ea340349f8848 from peer ae30e3bd7dc1421998cf9258aaa6bf7f ({username='slave'} at 127.0.0.1:35518)
I20240502 20:47:07.231575 26553 tablet_copy_service.cc:161] P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Beginning new tablet copy session on tablet ca0df961aea849f4bc2ea340349f8848 from peer ae30e3bd7dc1421998cf9258aaa6bf7f at {username='slave'} at 127.0.0.1:35518: session id = ae30e3bd7dc1421998cf9258aaa6bf7f-ca0df961aea849f4bc2ea340349f8848
I20240502 20:47:07.232172 27124 tablet_copy_client.cc:783] T 7d1ff41afa6e46609e6a3eb277df18ad P c8ef2de5cb854720b1090ff20c2fd13c: tablet copy: Starting download of 0 data blocks...
I20240502 20:47:07.232316 27124 tablet_copy_client.cc:647] T 7d1ff41afa6e46609e6a3eb277df18ad P c8ef2de5cb854720b1090ff20c2fd13c: tablet copy: Starting download of 1 WAL segments...
I20240502 20:47:07.232476 26553 tablet_copy_source_session.cc:205] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:47:07.232668 27131 tablet_copy_client.cc:783] T 8ea4b2412a3a420f93d384aa48117c60 P ae30e3bd7dc1421998cf9258aaa6bf7f: tablet copy: Starting download of 0 data blocks...
I20240502 20:47:07.232795 27131 tablet_copy_client.cc:647] T 8ea4b2412a3a420f93d384aa48117c60 P ae30e3bd7dc1421998cf9258aaa6bf7f: tablet copy: Starting download of 1 WAL segments...
I20240502 20:47:07.233212 27132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca0df961aea849f4bc2ea340349f8848. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:07.233861 27139 tablet_copy_client.cc:1002] T 8ea4b2412a3a420f93d384aa48117c60 P ae30e3bd7dc1421998cf9258aaa6bf7f: tablet copy: Injecting latency into file download: 2000
I20240502 20:47:07.234124 26390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1409e44c98f44bd888eea4cee4f1163a" candidate_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "97406d5f0d1f4d59bdc784684baa1559"
I20240502 20:47:07.234239 26390 raft_consensus.cc:3037] T 1409e44c98f44bd888eea4cee4f1163a P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:07.234351 27134 raft_consensus.cc:686] T 1409e44c98f44bd888eea4cee4f1163a P c8ef2de5cb854720b1090ff20c2fd13c [term 2 LEADER]: Becoming Leader. State: Replica: c8ef2de5cb854720b1090ff20c2fd13c, State: Running, Role: LEADER
I20240502 20:47:07.234489 27134 consensus_queue.cc:237] T 1409e44c98f44bd888eea4cee4f1163a P c8ef2de5cb854720b1090ff20c2fd13c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } }
I20240502 20:47:07.235098 27132 tablet_copy_client.cc:783] T ca0df961aea849f4bc2ea340349f8848 P ae30e3bd7dc1421998cf9258aaa6bf7f: tablet copy: Starting download of 0 data blocks...
I20240502 20:47:07.235229 27132 tablet_copy_client.cc:647] T ca0df961aea849f4bc2ea340349f8848 P ae30e3bd7dc1421998cf9258aaa6bf7f: tablet copy: Starting download of 1 WAL segments...
I20240502 20:47:07.235272 26390 raft_consensus.cc:2454] T 1409e44c98f44bd888eea4cee4f1163a P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8ef2de5cb854720b1090ff20c2fd13c in term 2.
I20240502 20:47:07.235711 27126 tablet_copy_client.cc:1002] T 7d1ff41afa6e46609e6a3eb277df18ad P c8ef2de5cb854720b1090ff20c2fd13c: tablet copy: Injecting latency into file download: 2000
I20240502 20:47:07.236649 26283 catalog_manager.cc:5484] T 1409e44c98f44bd888eea4cee4f1163a P c8ef2de5cb854720b1090ff20c2fd13c reported cstate change: term changed from 1 to 2, leader changed from bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131) to c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134). New cstate: current_term: 2 leader_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:47:07.237779 27133 tablet_copy_client.cc:1002] T ca0df961aea849f4bc2ea340349f8848 P ae30e3bd7dc1421998cf9258aaa6bf7f: tablet copy: Injecting latency into file download: 2000
I20240502 20:47:07.252647 27086 raft_consensus.cc:979] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830: : Instructing follower 2c2b4bb6fbe94bb8bea2eb59520f5fec to start an election
I20240502 20:47:07.252772 27086 raft_consensus.cc:1067] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 LEADER]: Signalling peer 2c2b4bb6fbe94bb8bea2eb59520f5fec to start an election
I20240502 20:47:07.253324 26860 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8ea4b2412a3a420f93d384aa48117c60"
dest_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec"
from {username='slave'} at 127.0.0.1:32794
I20240502 20:47:07.253495 26860 raft_consensus.cc:482] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:47:07.253564 26860 raft_consensus.cc:3037] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:07.254390 26860 raft_consensus.cc:504] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } }
I20240502 20:47:07.254885 26860 leader_election.cc:288] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [CANDIDATE]: Term 2 election: Requested vote from peers 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989), ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519)
I20240502 20:47:07.255157 26464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea4b2412a3a420f93d384aa48117c60" candidate_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830"
I20240502 20:47:07.255268 26464 raft_consensus.cc:3032] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:47:07.255306 26464 raft_consensus.cc:729] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ddb0f390a80f4c059c1a99a167c7d830, State: Running, Role: LEADER
I20240502 20:47:07.255437 26464 consensus_queue.cc:260] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } }
I20240502 20:47:07.255664 26464 raft_consensus.cc:3037] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:07.256704 26464 raft_consensus.cc:2454] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c2b4bb6fbe94bb8bea2eb59520f5fec in term 2.
I20240502 20:47:07.256973 26826 leader_election.cc:302] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c2b4bb6fbe94bb8bea2eb59520f5fec, ddb0f390a80f4c059c1a99a167c7d830; no voters:
I20240502 20:47:07.257181 27105 raft_consensus.cc:2781] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:47:07.257350 27105 raft_consensus.cc:686] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 2 LEADER]: Becoming Leader. State: Replica: 2c2b4bb6fbe94bb8bea2eb59520f5fec, State: Running, Role: LEADER
I20240502 20:47:07.257500 27105 consensus_queue.cc:237] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } }
I20240502 20:47:07.258924 26390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea4b2412a3a420f93d384aa48117c60" candidate_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "97406d5f0d1f4d59bdc784684baa1559"
I20240502 20:47:07.259032 26390 raft_consensus.cc:3037] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:07.259282 26283 catalog_manager.cc:5484] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec reported cstate change: term changed from 1 to 2, leader changed from ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130) to 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133). New cstate: current_term: 2 leader_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:07.259778 26390 raft_consensus.cc:2454] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c2b4bb6fbe94bb8bea2eb59520f5fec in term 2.
I20240502 20:47:07.301285 27148 ts_tablet_manager.cc:902] T 082557bf92e4443cbb9d0c50a89d24c4 P c8ef2de5cb854720b1090ff20c2fd13c: Initiating tablet copy from peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881)
I20240502 20:47:07.301858 27148 tablet_copy_client.cc:308] T 082557bf92e4443cbb9d0c50a89d24c4 P c8ef2de5cb854720b1090ff20c2fd13c: tablet copy: Beginning tablet copy session from remote peer at address 127.22.104.131:38881
I20240502 20:47:07.302495 26553 tablet_copy_service.cc:140] P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Received BeginTabletCopySession request for tablet 082557bf92e4443cbb9d0c50a89d24c4 from peer c8ef2de5cb854720b1090ff20c2fd13c ({username='slave'} at 127.0.0.1:49944)
I20240502 20:47:07.302594 26553 tablet_copy_service.cc:161] P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Beginning new tablet copy session on tablet 082557bf92e4443cbb9d0c50a89d24c4 from peer c8ef2de5cb854720b1090ff20c2fd13c at {username='slave'} at 127.0.0.1:49944: session id = c8ef2de5cb854720b1090ff20c2fd13c-082557bf92e4443cbb9d0c50a89d24c4
I20240502 20:47:07.303583 26553 tablet_copy_source_session.cc:205] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:47:07.304386 27148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 082557bf92e4443cbb9d0c50a89d24c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:07.306800 27148 tablet_copy_client.cc:783] T 082557bf92e4443cbb9d0c50a89d24c4 P c8ef2de5cb854720b1090ff20c2fd13c: tablet copy: Starting download of 0 data blocks...
I20240502 20:47:07.307020 27148 tablet_copy_client.cc:647] T 082557bf92e4443cbb9d0c50a89d24c4 P c8ef2de5cb854720b1090ff20c2fd13c: tablet copy: Starting download of 1 WAL segments...
I20240502 20:47:07.308328 27152 tablet_copy_client.cc:1002] T 082557bf92e4443cbb9d0c50a89d24c4 P c8ef2de5cb854720b1090ff20c2fd13c: tablet copy: Injecting latency into file download: 2000
I20240502 20:47:07.308401 27153 ts_tablet_manager.cc:902] T 7d7f28e11e1f42959251a7b03011f5dc P 2c2b4bb6fbe94bb8bea2eb59520f5fec: Initiating tablet copy from peer 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989)
I20240502 20:47:07.309092 27153 tablet_copy_client.cc:308] T 7d7f28e11e1f42959251a7b03011f5dc P 2c2b4bb6fbe94bb8bea2eb59520f5fec: tablet copy: Beginning tablet copy session from remote peer at address 127.22.104.129:41989
I20240502 20:47:07.309501 26401 tablet_copy_service.cc:140] P 97406d5f0d1f4d59bdc784684baa1559: Received BeginTabletCopySession request for tablet 7d7f28e11e1f42959251a7b03011f5dc from peer 2c2b4bb6fbe94bb8bea2eb59520f5fec ({username='slave'} at 127.0.0.1:45116)
I20240502 20:47:07.309597 26401 tablet_copy_service.cc:161] P 97406d5f0d1f4d59bdc784684baa1559: Beginning new tablet copy session on tablet 7d7f28e11e1f42959251a7b03011f5dc from peer 2c2b4bb6fbe94bb8bea2eb59520f5fec at {username='slave'} at 127.0.0.1:45116: session id = 2c2b4bb6fbe94bb8bea2eb59520f5fec-7d7f28e11e1f42959251a7b03011f5dc
I20240502 20:47:07.310431 26401 tablet_copy_source_session.cc:205] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:47:07.311239 27153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7f28e11e1f42959251a7b03011f5dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:07.313292 27153 tablet_copy_client.cc:783] T 7d7f28e11e1f42959251a7b03011f5dc P 2c2b4bb6fbe94bb8bea2eb59520f5fec: tablet copy: Starting download of 0 data blocks...
I20240502 20:47:07.313462 27153 tablet_copy_client.cc:647] T 7d7f28e11e1f42959251a7b03011f5dc P 2c2b4bb6fbe94bb8bea2eb59520f5fec: tablet copy: Starting download of 1 WAL segments...
I20240502 20:47:07.314608 27157 tablet_copy_client.cc:1002] T 7d7f28e11e1f42959251a7b03011f5dc P 2c2b4bb6fbe94bb8bea2eb59520f5fec: tablet copy: Injecting latency into file download: 2000
I20240502 20:47:07.355010 27084 raft_consensus.cc:979] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: : Instructing follower 97406d5f0d1f4d59bdc784684baa1559 to start an election
I20240502 20:47:07.355163 27084 raft_consensus.cc:1067] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 LEADER]: Signalling peer 97406d5f0d1f4d59bdc784684baa1559 to start an election
I20240502 20:47:07.355554 26390 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "082557bf92e4443cbb9d0c50a89d24c4"
dest_uuid: "97406d5f0d1f4d59bdc784684baa1559"
from {username='slave'} at 127.0.0.1:52166
I20240502 20:47:07.355708 26390 raft_consensus.cc:482] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:47:07.355777 26390 raft_consensus.cc:3037] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:07.356813 26390 raft_consensus.cc:504] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } }
I20240502 20:47:07.357205 26390 leader_election.cc:288] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [CANDIDATE]: Term 2 election: Requested vote from peers ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519), bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881)
I20240502 20:47:07.357436 26464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082557bf92e4443cbb9d0c50a89d24c4" candidate_uuid: "97406d5f0d1f4d59bdc784684baa1559" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830"
I20240502 20:47:07.357544 26464 raft_consensus.cc:3037] T 082557bf92e4443cbb9d0c50a89d24c4 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:07.357708 26538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082557bf92e4443cbb9d0c50a89d24c4" candidate_uuid: "97406d5f0d1f4d59bdc784684baa1559" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3"
I20240502 20:47:07.357823 26538 raft_consensus.cc:3032] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:47:07.357890 26538 raft_consensus.cc:729] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, State: Running, Role: LEADER
I20240502 20:47:07.358036 26538 consensus_queue.cc:260] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } }
I20240502 20:47:07.358227 26538 raft_consensus.cc:3037] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:07.358491 26464 raft_consensus.cc:2454] T 082557bf92e4443cbb9d0c50a89d24c4 P ddb0f390a80f4c059c1a99a167c7d830 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97406d5f0d1f4d59bdc784684baa1559 in term 2.
I20240502 20:47:07.358776 26353 leader_election.cc:302] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97406d5f0d1f4d59bdc784684baa1559, ddb0f390a80f4c059c1a99a167c7d830; no voters:
I20240502 20:47:07.358968 27055 raft_consensus.cc:2781] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:47:07.359058 27055 raft_consensus.cc:686] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [term 2 LEADER]: Becoming Leader. State: Replica: 97406d5f0d1f4d59bdc784684baa1559, State: Running, Role: LEADER
I20240502 20:47:07.359191 26538 raft_consensus.cc:2454] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97406d5f0d1f4d59bdc784684baa1559 in term 2.
I20240502 20:47:07.359184 27055 consensus_queue.cc:237] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } }
I20240502 20:47:07.360908 26283 catalog_manager.cc:5484] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 reported cstate change: term changed from 1 to 2, leader changed from bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131) to 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129). New cstate: current_term: 2 leader_uuid: "97406d5f0d1f4d59bdc784684baa1559" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 20:47:07.531136 26505 consensus_peers.cc:485] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 -> Peer 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133:43257): Couldn't send request to peer 2c2b4bb6fbe94bb8bea2eb59520f5fec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32631c89419245dea46abe6baa5dc295. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:07.552266 26391 raft_consensus.cc:1261] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 20:47:07.552696 27084 consensus_queue.cc:1035] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 20:47:07.554231 26980 raft_consensus.cc:979] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: : Instructing follower 97406d5f0d1f4d59bdc784684baa1559 to start an election
I20240502 20:47:07.554360 27084 raft_consensus.cc:1067] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 LEADER]: Signalling peer 97406d5f0d1f4d59bdc784684baa1559 to start an election
I20240502 20:47:07.554884 26391 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "32631c89419245dea46abe6baa5dc295"
dest_uuid: "97406d5f0d1f4d59bdc784684baa1559"
from {username='slave'} at 127.0.0.1:52166
I20240502 20:47:07.555261 26464 raft_consensus.cc:1261] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [term 2 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 20:47:07.555423 26391 raft_consensus.cc:482] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:47:07.555495 26391 raft_consensus.cc:3037] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:47:07.555583 27159 consensus_queue.cc:1035] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 20:47:07.556624 26391 raft_consensus.cc:504] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } }
I20240502 20:47:07.556993 26391 leader_election.cc:288] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [CANDIDATE]: Term 3 election: Requested vote from peers bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881), ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519)
I20240502 20:47:07.557245 26464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32631c89419245dea46abe6baa5dc295" candidate_uuid: "97406d5f0d1f4d59bdc784684baa1559" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830"
I20240502 20:47:07.557341 26464 raft_consensus.cc:3037] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:47:07.557659 26538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32631c89419245dea46abe6baa5dc295" candidate_uuid: "97406d5f0d1f4d59bdc784684baa1559" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3"
I20240502 20:47:07.557772 26538 raft_consensus.cc:3032] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 20:47:07.557812 26538 raft_consensus.cc:729] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, State: Running, Role: LEADER
I20240502 20:47:07.557914 26538 consensus_queue.cc:260] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } }
I20240502 20:47:07.558043 26538 raft_consensus.cc:3037] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:47:07.558229 26464 raft_consensus.cc:2454] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97406d5f0d1f4d59bdc784684baa1559 in term 3.
I20240502 20:47:07.558503 26353 leader_election.cc:302] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97406d5f0d1f4d59bdc784684baa1559, ddb0f390a80f4c059c1a99a167c7d830; no voters:
I20240502 20:47:07.558678 27055 raft_consensus.cc:2781] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 20:47:07.558789 27055 raft_consensus.cc:686] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [term 3 LEADER]: Becoming Leader. State: Replica: 97406d5f0d1f4d59bdc784684baa1559, State: Running, Role: LEADER
I20240502 20:47:07.558847 26538 raft_consensus.cc:2454] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97406d5f0d1f4d59bdc784684baa1559 in term 3.
I20240502 20:47:07.558903 27055 consensus_queue.cc:237] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } }
I20240502 20:47:07.561357 26279 catalog_manager.cc:5484] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 reported cstate change: term changed from 2 to 3, leader changed from bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131) to 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129). New cstate: current_term: 3 leader_uuid: "97406d5f0d1f4d59bdc784684baa1559" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:07.660406 26541 raft_consensus.cc:1261] T 23f60af679e74f078fa50af755a9fbf6 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 20:47:07.660880 27086 consensus_queue.cc:1035] T 23f60af679e74f078fa50af755a9fbf6 P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 20:47:07.663720 26391 raft_consensus.cc:1261] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 20:47:07.664206 27165 consensus_queue.cc:1035] T 23f60af679e74f078fa50af755a9fbf6 P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 20:47:07.681787 27113 raft_consensus.cc:979] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559: : Instructing follower ae30e3bd7dc1421998cf9258aaa6bf7f to start an election
I20240502 20:47:07.681901 27113 raft_consensus.cc:1067] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 LEADER]: Signalling peer ae30e3bd7dc1421998cf9258aaa6bf7f to start an election
I20240502 20:47:07.682271 26787 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "912fa9b5ccf74aadb519a8bf16659478"
dest_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f"
from {username='slave'} at 127.0.0.1:51670
I20240502 20:47:07.682399 26787 raft_consensus.cc:482] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:47:07.682488 26787 raft_consensus.cc:3037] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:07.683578 26787 raft_consensus.cc:504] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } }
I20240502 20:47:07.684015 26787 leader_election.cc:288] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f [CANDIDATE]: Term 2 election: Requested vote from peers 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989), bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881)
I20240502 20:47:07.684376 26391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "912fa9b5ccf74aadb519a8bf16659478" candidate_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "97406d5f0d1f4d59bdc784684baa1559"
I20240502 20:47:07.684387 26538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "912fa9b5ccf74aadb519a8bf16659478" candidate_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3"
I20240502 20:47:07.684485 26391 raft_consensus.cc:3032] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:47:07.684525 26538 raft_consensus.cc:3037] T 912fa9b5ccf74aadb519a8bf16659478 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:07.685315 26538 raft_consensus.cc:2454] T 912fa9b5ccf74aadb519a8bf16659478 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae30e3bd7dc1421998cf9258aaa6bf7f in term 2.
I20240502 20:47:07.684533 26391 raft_consensus.cc:729] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 97406d5f0d1f4d59bdc784684baa1559, State: Running, Role: LEADER
I20240502 20:47:07.685599 26751 leader_election.cc:302] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae30e3bd7dc1421998cf9258aaa6bf7f, bdcd7bbd5fa14f21bbf0a7a3cdc33cb3; no voters:
I20240502 20:47:07.685601 26391 consensus_queue.cc:260] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } }
I20240502 20:47:07.685825 26391 raft_consensus.cc:3037] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:07.685959 27167 raft_consensus.cc:2781] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:47:07.686149 27167 raft_consensus.cc:686] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f [term 2 LEADER]: Becoming Leader. State: Replica: ae30e3bd7dc1421998cf9258aaa6bf7f, State: Running, Role: LEADER
I20240502 20:47:07.686317 27167 consensus_queue.cc:237] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } }
I20240502 20:47:07.686738 26391 raft_consensus.cc:2454] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae30e3bd7dc1421998cf9258aaa6bf7f in term 2.
I20240502 20:47:07.688189 26279 catalog_manager.cc:5484] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f reported cstate change: term changed from 1 to 2, leader changed from 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129) to ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132). New cstate: current_term: 2 leader_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:47:07.760304 26464 raft_consensus.cc:1261] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [term 2 FOLLOWER]: Refusing update from remote peer 2c2b4bb6fbe94bb8bea2eb59520f5fec: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240502 20:47:07.760728 27171 consensus_queue.cc:1035] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 20:47:07.763407 26391 raft_consensus.cc:1261] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Refusing update from remote peer 2c2b4bb6fbe94bb8bea2eb59520f5fec: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240502 20:47:07.763756 27172 consensus_queue.cc:1035] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
W20240502 20:47:07.768234 26829 consensus_peers.cc:485] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec -> Peer ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132:37635): Couldn't send request to peer ae30e3bd7dc1421998cf9258aaa6bf7f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:07.818198 26538 raft_consensus.cc:1261] T 1409e44c98f44bd888eea4cee4f1163a P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 FOLLOWER]: Refusing update from remote peer c8ef2de5cb854720b1090ff20c2fd13c: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240502 20:47:07.818732 27176 consensus_queue.cc:1035] T 1409e44c98f44bd888eea4cee4f1163a P c8ef2de5cb854720b1090ff20c2fd13c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 20:47:07.821599 26391 raft_consensus.cc:1261] T 1409e44c98f44bd888eea4cee4f1163a P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Refusing update from remote peer c8ef2de5cb854720b1090ff20c2fd13c: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240502 20:47:07.822465 27176 consensus_queue.cc:1035] T 1409e44c98f44bd888eea4cee4f1163a P c8ef2de5cb854720b1090ff20c2fd13c [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
W20240502 20:47:07.856382 26355 consensus_peers.cc:485] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 -> Peer c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134:45157): Couldn't send request to peer c8ef2de5cb854720b1090ff20c2fd13c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:07.868104 26538 raft_consensus.cc:1261] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 20:47:07.868620 27170 consensus_queue.cc:1035] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 20:47:07.871694 26461 raft_consensus.cc:1261] T 082557bf92e4443cbb9d0c50a89d24c4 P ddb0f390a80f4c059c1a99a167c7d830 [term 2 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 20:47:07.872084 27170 consensus_queue.cc:1035] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
W20240502 20:47:08.048537 26353 consensus_peers.cc:485] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 -> Peer 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133:43257): Couldn't send request to peer 2c2b4bb6fbe94bb8bea2eb59520f5fec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32631c89419245dea46abe6baa5dc295. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:08.068733 26538 raft_consensus.cc:1261] T 912fa9b5ccf74aadb519a8bf16659478 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 FOLLOWER]: Refusing update from remote peer ae30e3bd7dc1421998cf9258aaa6bf7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 20:47:08.069139 27167 consensus_queue.cc:1035] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 20:47:08.072062 26391 raft_consensus.cc:1261] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Refusing update from remote peer ae30e3bd7dc1421998cf9258aaa6bf7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 20:47:08.072417 27189 consensus_queue.cc:1035] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 20:47:08.089953 26541 raft_consensus.cc:1261] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 3 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 20:47:08.090345 27170 consensus_queue.cc:1035] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 20:47:08.092589 26461 raft_consensus.cc:1261] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [term 3 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 20:47:08.093165 27055 consensus_queue.cc:1035] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 20:47:08.439914 27195 ts_tablet_manager.cc:902] T 32631c89419245dea46abe6baa5dc295 P 2c2b4bb6fbe94bb8bea2eb59520f5fec: Initiating tablet copy from peer 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989)
I20240502 20:47:08.440497 27195 tablet_copy_client.cc:308] T 32631c89419245dea46abe6baa5dc295 P 2c2b4bb6fbe94bb8bea2eb59520f5fec: tablet copy: Beginning tablet copy session from remote peer at address 127.22.104.129:41989
I20240502 20:47:08.440904 26401 tablet_copy_service.cc:140] P 97406d5f0d1f4d59bdc784684baa1559: Received BeginTabletCopySession request for tablet 32631c89419245dea46abe6baa5dc295 from peer 2c2b4bb6fbe94bb8bea2eb59520f5fec ({username='slave'} at 127.0.0.1:45116)
I20240502 20:47:08.441007 26401 tablet_copy_service.cc:161] P 97406d5f0d1f4d59bdc784684baa1559: Beginning new tablet copy session on tablet 32631c89419245dea46abe6baa5dc295 from peer 2c2b4bb6fbe94bb8bea2eb59520f5fec at {username='slave'} at 127.0.0.1:45116: session id = 2c2b4bb6fbe94bb8bea2eb59520f5fec-32631c89419245dea46abe6baa5dc295
I20240502 20:47:08.442034 26401 tablet_copy_source_session.cc:205] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:47:08.442914 27195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32631c89419245dea46abe6baa5dc295. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:08.445209 27195 tablet_copy_client.cc:783] T 32631c89419245dea46abe6baa5dc295 P 2c2b4bb6fbe94bb8bea2eb59520f5fec: tablet copy: Starting download of 0 data blocks...
I20240502 20:47:08.445442 27195 tablet_copy_client.cc:647] T 32631c89419245dea46abe6baa5dc295 P 2c2b4bb6fbe94bb8bea2eb59520f5fec: tablet copy: Starting download of 1 WAL segments...
I20240502 20:47:08.446360 27197 tablet_copy_client.cc:1002] T 32631c89419245dea46abe6baa5dc295 P 2c2b4bb6fbe94bb8bea2eb59520f5fec: tablet copy: Injecting latency into file download: 2000
I20240502 20:47:09.126616 26341 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:47:09.128623 26391 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "082557bf92e4443cbb9d0c50a89d24c4"
dest_uuid: "97406d5f0d1f4d59bdc784684baa1559"
mode: GRACEFUL
new_leader_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3"
from {username='slave'} at 127.0.0.1:45110
I20240502 20:47:09.128799 26391 raft_consensus.cc:595] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [term 2 LEADER]: Received request to transfer leadership to TS bdcd7bbd5fa14f21bbf0a7a3cdc33cb3
I20240502 20:47:09.129514 26461 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5db41d5853a34dc594bafb0f4c53cd7d"
dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830"
mode: GRACEFUL
new_leader_uuid: "97406d5f0d1f4d59bdc784684baa1559"
from {username='slave'} at 127.0.0.1:43758
I20240502 20:47:09.129638 26461 raft_consensus.cc:595] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [term 1 LEADER]: Received request to transfer leadership to TS 97406d5f0d1f4d59bdc784684baa1559
I20240502 20:47:09.130256 26391 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7d7f28e11e1f42959251a7b03011f5dc"
dest_uuid: "97406d5f0d1f4d59bdc784684baa1559"
mode: GRACEFUL
new_leader_uuid: "ddb0f390a80f4c059c1a99a167c7d830"
from {username='slave'} at 127.0.0.1:45110
I20240502 20:47:09.130355 26391 raft_consensus.cc:595] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [term 1 LEADER]: Received request to transfer leadership to TS ddb0f390a80f4c059c1a99a167c7d830
I20240502 20:47:09.130586 26341 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240502 20:47:09.130750 26341 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 20:47:09.193275 27113 raft_consensus.cc:979] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559: : Instructing follower bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 to start an election
I20240502 20:47:09.193467 27170 raft_consensus.cc:1067] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [term 2 LEADER]: Signalling peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 to start an election
I20240502 20:47:09.193861 26541 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "082557bf92e4443cbb9d0c50a89d24c4"
dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3"
from {username='slave'} at 127.0.0.1:49898
I20240502 20:47:09.193984 26541 raft_consensus.cc:482] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:47:09.194049 26541 raft_consensus.cc:3037] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:47:09.195186 26541 raft_consensus.cc:504] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } }
I20240502 20:47:09.195569 26541 leader_election.cc:288] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [CANDIDATE]: Term 3 election: Requested vote from peers 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989), ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519)
I20240502 20:47:09.195775 26390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082557bf92e4443cbb9d0c50a89d24c4" candidate_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "97406d5f0d1f4d59bdc784684baa1559"
I20240502 20:47:09.195858 26461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082557bf92e4443cbb9d0c50a89d24c4" candidate_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830"
I20240502 20:47:09.195897 26390 raft_consensus.cc:3032] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 20:47:09.195955 26461 raft_consensus.cc:3037] T 082557bf92e4443cbb9d0c50a89d24c4 P ddb0f390a80f4c059c1a99a167c7d830 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:47:09.195971 26390 raft_consensus.cc:729] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 97406d5f0d1f4d59bdc784684baa1559, State: Running, Role: LEADER
I20240502 20:47:09.196105 26390 consensus_queue.cc:260] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } }
I20240502 20:47:09.196425 26390 raft_consensus.cc:3037] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:47:09.196847 26461 raft_consensus.cc:2454] T 082557bf92e4443cbb9d0c50a89d24c4 P ddb0f390a80f4c059c1a99a167c7d830 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 in term 3.
I20240502 20:47:09.197124 26505 leader_election.cc:302] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, ddb0f390a80f4c059c1a99a167c7d830; no voters:
I20240502 20:47:09.197450 26980 raft_consensus.cc:2781] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 20:47:09.197618 26980 raft_consensus.cc:686] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 3 LEADER]: Becoming Leader. State: Replica: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, State: Running, Role: LEADER
I20240502 20:47:09.197623 26390 raft_consensus.cc:2454] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 in term 3.
I20240502 20:47:09.197767 26980 consensus_queue.cc:237] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } }
I20240502 20:47:09.199287 26283 catalog_manager.cc:5484] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 reported cstate change: term changed from 2 to 3, leader changed from 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129) to bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131). New cstate: current_term: 3 leader_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:09.235417 27131 tablet_copy_client.cc:523] T 8ea4b2412a3a420f93d384aa48117c60 P ae30e3bd7dc1421998cf9258aaa6bf7f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:47:09.236207 27124 tablet_copy_client.cc:523] T 7d1ff41afa6e46609e6a3eb277df18ad P c8ef2de5cb854720b1090ff20c2fd13c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:47:09.236935 27131 tablet_bootstrap.cc:492] T 8ea4b2412a3a420f93d384aa48117c60 P ae30e3bd7dc1421998cf9258aaa6bf7f: Bootstrap starting.
I20240502 20:47:09.237538 27124 tablet_bootstrap.cc:492] T 7d1ff41afa6e46609e6a3eb277df18ad P c8ef2de5cb854720b1090ff20c2fd13c: Bootstrap starting.
I20240502 20:47:09.239631 27132 tablet_copy_client.cc:523] T ca0df961aea849f4bc2ea340349f8848 P ae30e3bd7dc1421998cf9258aaa6bf7f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:47:09.240890 27132 tablet_bootstrap.cc:492] T ca0df961aea849f4bc2ea340349f8848 P ae30e3bd7dc1421998cf9258aaa6bf7f: Bootstrap starting.
I20240502 20:47:09.261790 27131 tablet_bootstrap.cc:492] T 8ea4b2412a3a420f93d384aa48117c60 P ae30e3bd7dc1421998cf9258aaa6bf7f: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=113 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:47:09.262240 27131 tablet_bootstrap.cc:492] T 8ea4b2412a3a420f93d384aa48117c60 P ae30e3bd7dc1421998cf9258aaa6bf7f: Bootstrap complete.
I20240502 20:47:09.262565 27131 ts_tablet_manager.cc:1367] T 8ea4b2412a3a420f93d384aa48117c60 P ae30e3bd7dc1421998cf9258aaa6bf7f: Time spent bootstrapping tablet: real 0.026s user 0.026s sys 0.000s
I20240502 20:47:09.263250 27131 raft_consensus.cc:348] T 8ea4b2412a3a420f93d384aa48117c60 P ae30e3bd7dc1421998cf9258aaa6bf7f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } }
I20240502 20:47:09.263447 27131 raft_consensus.cc:729] T 8ea4b2412a3a420f93d384aa48117c60 P ae30e3bd7dc1421998cf9258aaa6bf7f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ae30e3bd7dc1421998cf9258aaa6bf7f, State: Initialized, Role: LEARNER
I20240502 20:47:09.263576 27131 consensus_queue.cc:260] T 8ea4b2412a3a420f93d384aa48117c60 P ae30e3bd7dc1421998cf9258aaa6bf7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } }
I20240502 20:47:09.263916 27131 ts_tablet_manager.cc:1398] T 8ea4b2412a3a420f93d384aa48117c60 P ae30e3bd7dc1421998cf9258aaa6bf7f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:47:09.263346 27124 tablet_bootstrap.cc:492] T 7d1ff41afa6e46609e6a3eb277df18ad P c8ef2de5cb854720b1090ff20c2fd13c: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=121 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:47:09.264210 27124 tablet_bootstrap.cc:492] T 7d1ff41afa6e46609e6a3eb277df18ad P c8ef2de5cb854720b1090ff20c2fd13c: Bootstrap complete.
I20240502 20:47:09.264575 27124 ts_tablet_manager.cc:1367] T 7d1ff41afa6e46609e6a3eb277df18ad P c8ef2de5cb854720b1090ff20c2fd13c: Time spent bootstrapping tablet: real 0.027s user 0.027s sys 0.000s
I20240502 20:47:09.264601 26476 tablet_copy_service.cc:342] P ddb0f390a80f4c059c1a99a167c7d830: Request end of tablet copy session ae30e3bd7dc1421998cf9258aaa6bf7f-8ea4b2412a3a420f93d384aa48117c60 received from {username='slave'} at 127.0.0.1:56924
I20240502 20:47:09.264711 26476 tablet_copy_service.cc:433] P ddb0f390a80f4c059c1a99a167c7d830: ending tablet copy session ae30e3bd7dc1421998cf9258aaa6bf7f-8ea4b2412a3a420f93d384aa48117c60 on tablet 8ea4b2412a3a420f93d384aa48117c60 with peer ae30e3bd7dc1421998cf9258aaa6bf7f
I20240502 20:47:09.265419 27124 raft_consensus.cc:348] T 7d1ff41afa6e46609e6a3eb277df18ad P c8ef2de5cb854720b1090ff20c2fd13c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } }
I20240502 20:47:09.265605 27124 raft_consensus.cc:729] T 7d1ff41afa6e46609e6a3eb277df18ad P c8ef2de5cb854720b1090ff20c2fd13c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8ef2de5cb854720b1090ff20c2fd13c, State: Initialized, Role: LEARNER
I20240502 20:47:09.265753 27124 consensus_queue.cc:260] T 7d1ff41afa6e46609e6a3eb277df18ad P c8ef2de5cb854720b1090ff20c2fd13c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 1.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } }
I20240502 20:47:09.266104 27124 ts_tablet_manager.cc:1398] T 7d1ff41afa6e46609e6a3eb277df18ad P c8ef2de5cb854720b1090ff20c2fd13c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:47:09.266669 26476 tablet_copy_service.cc:342] P ddb0f390a80f4c059c1a99a167c7d830: Request end of tablet copy session c8ef2de5cb854720b1090ff20c2fd13c-7d1ff41afa6e46609e6a3eb277df18ad received from {username='slave'} at 127.0.0.1:43774
I20240502 20:47:09.266765 26476 tablet_copy_service.cc:433] P ddb0f390a80f4c059c1a99a167c7d830: ending tablet copy session c8ef2de5cb854720b1090ff20c2fd13c-7d1ff41afa6e46609e6a3eb277df18ad on tablet 7d1ff41afa6e46609e6a3eb277df18ad with peer c8ef2de5cb854720b1090ff20c2fd13c
I20240502 20:47:09.278155 27132 tablet_bootstrap.cc:492] T ca0df961aea849f4bc2ea340349f8848 P ae30e3bd7dc1421998cf9258aaa6bf7f: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=94 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:47:09.278486 27132 tablet_bootstrap.cc:492] T ca0df961aea849f4bc2ea340349f8848 P ae30e3bd7dc1421998cf9258aaa6bf7f: Bootstrap complete.
I20240502 20:47:09.278682 27132 ts_tablet_manager.cc:1367] T ca0df961aea849f4bc2ea340349f8848 P ae30e3bd7dc1421998cf9258aaa6bf7f: Time spent bootstrapping tablet: real 0.038s user 0.020s sys 0.000s
I20240502 20:47:09.279188 27132 raft_consensus.cc:348] T ca0df961aea849f4bc2ea340349f8848 P ae30e3bd7dc1421998cf9258aaa6bf7f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } }
I20240502 20:47:09.279326 27132 raft_consensus.cc:729] T ca0df961aea849f4bc2ea340349f8848 P ae30e3bd7dc1421998cf9258aaa6bf7f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ae30e3bd7dc1421998cf9258aaa6bf7f, State: Initialized, Role: LEARNER
I20240502 20:47:09.279431 27132 consensus_queue.cc:260] T ca0df961aea849f4bc2ea340349f8848 P ae30e3bd7dc1421998cf9258aaa6bf7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } }
I20240502 20:47:09.279791 27132 ts_tablet_manager.cc:1398] T ca0df961aea849f4bc2ea340349f8848 P ae30e3bd7dc1421998cf9258aaa6bf7f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:47:09.280287 26553 tablet_copy_service.cc:342] P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Request end of tablet copy session ae30e3bd7dc1421998cf9258aaa6bf7f-ca0df961aea849f4bc2ea340349f8848 received from {username='slave'} at 127.0.0.1:35518
I20240502 20:47:09.280364 26553 tablet_copy_service.cc:433] P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: ending tablet copy session ae30e3bd7dc1421998cf9258aaa6bf7f-ca0df961aea849f4bc2ea340349f8848 on tablet ca0df961aea849f4bc2ea340349f8848 with peer ae30e3bd7dc1421998cf9258aaa6bf7f
I20240502 20:47:09.300880 27165 raft_consensus.cc:979] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830: : Instructing follower 97406d5f0d1f4d59bdc784684baa1559 to start an election
I20240502 20:47:09.300990 27165 raft_consensus.cc:1067] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [term 1 LEADER]: Signalling peer 97406d5f0d1f4d59bdc784684baa1559 to start an election
I20240502 20:47:09.301370 26390 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5db41d5853a34dc594bafb0f4c53cd7d"
dest_uuid: "97406d5f0d1f4d59bdc784684baa1559"
from {username='slave'} at 127.0.0.1:52170
I20240502 20:47:09.301506 26390 raft_consensus.cc:482] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:47:09.301565 26390 raft_consensus.cc:3037] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:09.302359 26390 raft_consensus.cc:504] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } }
I20240502 20:47:09.302770 26390 leader_election.cc:288] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [CANDIDATE]: Term 2 election: Requested vote from peers ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519), ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132:37635)
I20240502 20:47:09.303023 26787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5db41d5853a34dc594bafb0f4c53cd7d" candidate_uuid: "97406d5f0d1f4d59bdc784684baa1559" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f"
I20240502 20:47:09.303179 26787 raft_consensus.cc:3037] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:09.303917 26787 raft_consensus.cc:2454] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97406d5f0d1f4d59bdc784684baa1559 in term 2.
I20240502 20:47:09.304203 26356 leader_election.cc:302] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97406d5f0d1f4d59bdc784684baa1559, ae30e3bd7dc1421998cf9258aaa6bf7f; no voters:
I20240502 20:47:09.304220 26464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5db41d5853a34dc594bafb0f4c53cd7d" candidate_uuid: "97406d5f0d1f4d59bdc784684baa1559" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830"
I20240502 20:47:09.304328 26464 raft_consensus.cc:3032] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:47:09.304373 26464 raft_consensus.cc:729] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ddb0f390a80f4c059c1a99a167c7d830, State: Running, Role: LEADER
I20240502 20:47:09.304390 27170 raft_consensus.cc:2781] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:47:09.304466 27170 raft_consensus.cc:686] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [term 2 LEADER]: Becoming Leader. State: Replica: 97406d5f0d1f4d59bdc784684baa1559, State: Running, Role: LEADER
I20240502 20:47:09.304543 26464 consensus_queue.cc:260] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } }
I20240502 20:47:09.304579 27170 consensus_queue.cc:237] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } }
I20240502 20:47:09.304725 26464 raft_consensus.cc:3037] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:09.305703 26464 raft_consensus.cc:2454] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97406d5f0d1f4d59bdc784684baa1559 in term 2.
I20240502 20:47:09.306362 26279 catalog_manager.cc:5484] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 reported cstate change: term changed from 1 to 2, leader changed from ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130) to 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129). New cstate: current_term: 2 leader_uuid: "97406d5f0d1f4d59bdc784684baa1559" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:09.308916 27148 tablet_copy_client.cc:523] T 082557bf92e4443cbb9d0c50a89d24c4 P c8ef2de5cb854720b1090ff20c2fd13c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:47:09.310017 27148 tablet_bootstrap.cc:492] T 082557bf92e4443cbb9d0c50a89d24c4 P c8ef2de5cb854720b1090ff20c2fd13c: Bootstrap starting.
I20240502 20:47:09.315390 27153 tablet_copy_client.cc:523] T 7d7f28e11e1f42959251a7b03011f5dc P 2c2b4bb6fbe94bb8bea2eb59520f5fec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:47:09.316838 27153 tablet_bootstrap.cc:492] T 7d7f28e11e1f42959251a7b03011f5dc P 2c2b4bb6fbe94bb8bea2eb59520f5fec: Bootstrap starting.
I20240502 20:47:09.329684 26787 raft_consensus.cc:3037] T 8ea4b2412a3a420f93d384aa48117c60 P ae30e3bd7dc1421998cf9258aaa6bf7f [term 1 LEARNER]: Advancing to term 2
I20240502 20:47:09.335608 27148 tablet_bootstrap.cc:492] T 082557bf92e4443cbb9d0c50a89d24c4 P c8ef2de5cb854720b1090ff20c2fd13c: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=123 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:47:09.336028 27148 tablet_bootstrap.cc:492] T 082557bf92e4443cbb9d0c50a89d24c4 P c8ef2de5cb854720b1090ff20c2fd13c: Bootstrap complete.
I20240502 20:47:09.336304 27148 ts_tablet_manager.cc:1367] T 082557bf92e4443cbb9d0c50a89d24c4 P c8ef2de5cb854720b1090ff20c2fd13c: Time spent bootstrapping tablet: real 0.026s user 0.027s sys 0.000s
I20240502 20:47:09.336823 27148 raft_consensus.cc:348] T 082557bf92e4443cbb9d0c50a89d24c4 P c8ef2de5cb854720b1090ff20c2fd13c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } }
I20240502 20:47:09.337055 27148 raft_consensus.cc:729] T 082557bf92e4443cbb9d0c50a89d24c4 P c8ef2de5cb854720b1090ff20c2fd13c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8ef2de5cb854720b1090ff20c2fd13c, State: Initialized, Role: LEARNER
I20240502 20:47:09.337236 27148 consensus_queue.cc:260] T 082557bf92e4443cbb9d0c50a89d24c4 P c8ef2de5cb854720b1090ff20c2fd13c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } }
I20240502 20:47:09.337707 27148 ts_tablet_manager.cc:1398] T 082557bf92e4443cbb9d0c50a89d24c4 P c8ef2de5cb854720b1090ff20c2fd13c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:47:09.338232 26553 tablet_copy_service.cc:342] P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Request end of tablet copy session c8ef2de5cb854720b1090ff20c2fd13c-082557bf92e4443cbb9d0c50a89d24c4 received from {username='slave'} at 127.0.0.1:49944
I20240502 20:47:09.338313 26553 tablet_copy_service.cc:433] P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: ending tablet copy session c8ef2de5cb854720b1090ff20c2fd13c-082557bf92e4443cbb9d0c50a89d24c4 on tablet 082557bf92e4443cbb9d0c50a89d24c4 with peer c8ef2de5cb854720b1090ff20c2fd13c
I20240502 20:47:09.344820 27153 tablet_bootstrap.cc:492] T 7d7f28e11e1f42959251a7b03011f5dc P 2c2b4bb6fbe94bb8bea2eb59520f5fec: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=113 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:47:09.345129 27153 tablet_bootstrap.cc:492] T 7d7f28e11e1f42959251a7b03011f5dc P 2c2b4bb6fbe94bb8bea2eb59520f5fec: Bootstrap complete.
I20240502 20:47:09.345326 27153 ts_tablet_manager.cc:1367] T 7d7f28e11e1f42959251a7b03011f5dc P 2c2b4bb6fbe94bb8bea2eb59520f5fec: Time spent bootstrapping tablet: real 0.029s user 0.023s sys 0.000s
I20240502 20:47:09.345810 27153 raft_consensus.cc:348] T 7d7f28e11e1f42959251a7b03011f5dc P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } }
I20240502 20:47:09.345955 27153 raft_consensus.cc:729] T 7d7f28e11e1f42959251a7b03011f5dc P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2c2b4bb6fbe94bb8bea2eb59520f5fec, State: Initialized, Role: LEARNER
I20240502 20:47:09.346071 27153 consensus_queue.cc:260] T 7d7f28e11e1f42959251a7b03011f5dc P 2c2b4bb6fbe94bb8bea2eb59520f5fec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } }
I20240502 20:47:09.346370 27153 ts_tablet_manager.cc:1398] T 7d7f28e11e1f42959251a7b03011f5dc P 2c2b4bb6fbe94bb8bea2eb59520f5fec: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:47:09.346827 26401 tablet_copy_service.cc:342] P 97406d5f0d1f4d59bdc784684baa1559: Request end of tablet copy session 2c2b4bb6fbe94bb8bea2eb59520f5fec-7d7f28e11e1f42959251a7b03011f5dc received from {username='slave'} at 127.0.0.1:45116
I20240502 20:47:09.346911 26401 tablet_copy_service.cc:433] P 97406d5f0d1f4d59bdc784684baa1559: ending tablet copy session 2c2b4bb6fbe94bb8bea2eb59520f5fec-7d7f28e11e1f42959251a7b03011f5dc on tablet 7d7f28e11e1f42959251a7b03011f5dc with peer 2c2b4bb6fbe94bb8bea2eb59520f5fec
I20240502 20:47:09.393311 26787 raft_consensus.cc:1203] T ca0df961aea849f4bc2ea340349f8848 P ae30e3bd7dc1421998cf9258aaa6bf7f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240502 20:47:09.499372 27113 raft_consensus.cc:979] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559: : Instructing follower ddb0f390a80f4c059c1a99a167c7d830 to start an election
I20240502 20:47:09.499505 27113 raft_consensus.cc:1067] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [term 1 LEADER]: Signalling peer ddb0f390a80f4c059c1a99a167c7d830 to start an election
I20240502 20:47:09.499899 26464 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7d7f28e11e1f42959251a7b03011f5dc"
dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830"
from {username='slave'} at 127.0.0.1:56890
I20240502 20:47:09.500037 26464 raft_consensus.cc:482] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:47:09.500103 26464 raft_consensus.cc:3037] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:09.501204 26464 raft_consensus.cc:504] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } }
I20240502 20:47:09.501602 26464 leader_election.cc:288] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [CANDIDATE]: Term 2 election: Requested vote from peers 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989), bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881)
I20240502 20:47:09.501806 26391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7f28e11e1f42959251a7b03011f5dc" candidate_uuid: "ddb0f390a80f4c059c1a99a167c7d830" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "97406d5f0d1f4d59bdc784684baa1559"
I20240502 20:47:09.501889 26538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7f28e11e1f42959251a7b03011f5dc" candidate_uuid: "ddb0f390a80f4c059c1a99a167c7d830" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3"
I20240502 20:47:09.501922 26391 raft_consensus.cc:3032] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:47:09.502013 26538 raft_consensus.cc:3037] T 7d7f28e11e1f42959251a7b03011f5dc P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:09.502038 26391 raft_consensus.cc:729] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 97406d5f0d1f4d59bdc784684baa1559, State: Running, Role: LEADER
I20240502 20:47:09.502197 26391 consensus_queue.cc:260] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } }
I20240502 20:47:09.502384 26391 raft_consensus.cc:3037] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:09.503031 26538 raft_consensus.cc:2454] T 7d7f28e11e1f42959251a7b03011f5dc P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddb0f390a80f4c059c1a99a167c7d830 in term 2.
I20240502 20:47:09.503197 26391 raft_consensus.cc:2454] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddb0f390a80f4c059c1a99a167c7d830 in term 2.
I20240502 20:47:09.503340 26431 leader_election.cc:302] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, ddb0f390a80f4c059c1a99a167c7d830; no voters:
I20240502 20:47:09.503489 27209 raft_consensus.cc:2781] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:47:09.503572 27209 raft_consensus.cc:686] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [term 2 LEADER]: Becoming Leader. State: Replica: ddb0f390a80f4c059c1a99a167c7d830, State: Running, Role: LEADER
I20240502 20:47:09.503693 27209 consensus_queue.cc:237] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } }
I20240502 20:47:09.505352 26279 catalog_manager.cc:5484] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 reported cstate change: term changed from 1 to 2, leader changed from 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129) to ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130). New cstate: current_term: 2 leader_uuid: "ddb0f390a80f4c059c1a99a167c7d830" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:09.510067 26464 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7d7f28e11e1f42959251a7b03011f5dc"
dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56876
I20240502 20:47:09.510195 26464 raft_consensus.cc:595] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [term 2 LEADER]: Received request to transfer leadership
I20240502 20:47:09.514066 26464 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7d7f28e11e1f42959251a7b03011f5dc"
dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56876
I20240502 20:47:09.514194 26464 raft_consensus.cc:595] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [term 2 LEADER]: Received request to transfer leadership
I20240502 20:47:09.515023 26464 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7d7f28e11e1f42959251a7b03011f5dc"
dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56876
I20240502 20:47:09.515132 26464 raft_consensus.cc:595] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [term 2 LEADER]: Received request to transfer leadership
W20240502 20:47:09.515421 26335 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 7d7f28e11e1f42959251a7b03011f5dc already in progress
W20240502 20:47:09.515468 26335 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 7d7f28e11e1f42959251a7b03011f5dc already in progress
I20240502 20:47:09.574159 26391 raft_consensus.cc:1261] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [term 3 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 20:47:09.574610 26980 consensus_queue.cc:1035] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 20:47:09.579437 26939 raft_consensus.cc:3037] T 082557bf92e4443cbb9d0c50a89d24c4 P c8ef2de5cb854720b1090ff20c2fd13c [term 1 LEARNER]: Advancing to term 3
I20240502 20:47:09.579540 26464 raft_consensus.cc:1261] T 082557bf92e4443cbb9d0c50a89d24c4 P ddb0f390a80f4c059c1a99a167c7d830 [term 3 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 20:47:09.580134 27200 consensus_queue.cc:1035] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 20:47:09.580698 26939 raft_consensus.cc:1261] T 082557bf92e4443cbb9d0c50a89d24c4 P c8ef2de5cb854720b1090ff20c2fd13c [term 3 LEARNER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 20:47:09.581378 27200 consensus_queue.cc:1035] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 20:47:09.583829 27200 raft_consensus.cc:1050] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: attempting to promote NON_VOTER c8ef2de5cb854720b1090ff20c2fd13c to VOTER
I20240502 20:47:09.584203 27200 consensus_queue.cc:237] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 30, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } }
I20240502 20:47:09.585309 26464 raft_consensus.cc:1261] T 082557bf92e4443cbb9d0c50a89d24c4 P ddb0f390a80f4c059c1a99a167c7d830 [term 3 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240502 20:47:09.585439 26939 raft_consensus.cc:1261] T 082557bf92e4443cbb9d0c50a89d24c4 P c8ef2de5cb854720b1090ff20c2fd13c [term 3 LEARNER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240502 20:47:09.585455 26391 raft_consensus.cc:1261] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [term 3 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240502 20:47:09.585691 27221 consensus_queue.cc:1035] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 20:47:09.585986 27229 consensus_queue.cc:1035] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 20:47:09.586158 27221 consensus_queue.cc:1035] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 20:47:09.587679 26980 raft_consensus.cc:2932] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 29 to 32, c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:09.588037 26391 raft_consensus.cc:2932] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 29 to 32, c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:09.590003 26461 raft_consensus.cc:2932] T 082557bf92e4443cbb9d0c50a89d24c4 P ddb0f390a80f4c059c1a99a167c7d830 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 29 to 32, c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:09.590183 26279 catalog_manager.cc:5484] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 reported cstate change: config changed from index 29 to 32, c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:09.590476 26939 raft_consensus.cc:2932] T 082557bf92e4443cbb9d0c50a89d24c4 P c8ef2de5cb854720b1090ff20c2fd13c [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 29 to 32, c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:09.594450 26538 consensus_queue.cc:237] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } }
I20240502 20:47:09.595861 26939 raft_consensus.cc:1261] T 082557bf92e4443cbb9d0c50a89d24c4 P c8ef2de5cb854720b1090ff20c2fd13c [term 3 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240502 20:47:09.595896 26391 raft_consensus.cc:1261] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [term 3 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240502 20:47:09.596218 27230 consensus_queue.cc:1035] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 20:47:09.596428 27229 consensus_queue.cc:1035] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 20:47:09.597690 27230 raft_consensus.cc:2932] T 082557bf92e4443cbb9d0c50a89d24c4 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 3 LEADER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:09.598202 26939 raft_consensus.cc:2932] T 082557bf92e4443cbb9d0c50a89d24c4 P c8ef2de5cb854720b1090ff20c2fd13c [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:09.599251 26272 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 082557bf92e4443cbb9d0c50a89d24c4 with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:47:09.598202 26391 raft_consensus.cc:2932] T 082557bf92e4443cbb9d0c50a89d24c4 P 97406d5f0d1f4d59bdc784684baa1559 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:09.600180 26283 catalog_manager.cc:5484] T 082557bf92e4443cbb9d0c50a89d24c4 P c8ef2de5cb854720b1090ff20c2fd13c reported cstate change: config changed from index 32 to 33, VOTER ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130) evicted. New cstate: current_term: 3 leader_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:09.602411 26449 tablet_service.cc:1514] Processing DeleteTablet for tablet 082557bf92e4443cbb9d0c50a89d24c4 with delete_type TABLET_DATA_TOMBSTONED (TS ddb0f390a80f4c059c1a99a167c7d830 not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:56876
I20240502 20:47:09.602843 27231 tablet_replica.cc:330] T 082557bf92e4443cbb9d0c50a89d24c4 P ddb0f390a80f4c059c1a99a167c7d830: stopping tablet replica
I20240502 20:47:09.603049 27231 raft_consensus.cc:2229] T 082557bf92e4443cbb9d0c50a89d24c4 P ddb0f390a80f4c059c1a99a167c7d830 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:09.603279 27231 raft_consensus.cc:2258] T 082557bf92e4443cbb9d0c50a89d24c4 P ddb0f390a80f4c059c1a99a167c7d830 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:09.603848 27231 ts_tablet_manager.cc:1875] T 082557bf92e4443cbb9d0c50a89d24c4 P ddb0f390a80f4c059c1a99a167c7d830: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:09.605926 27231 ts_tablet_manager.cc:1888] T 082557bf92e4443cbb9d0c50a89d24c4 P ddb0f390a80f4c059c1a99a167c7d830: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.32
I20240502 20:47:09.606030 27231 log.cc:1196] T 082557bf92e4443cbb9d0c50a89d24c4 P ddb0f390a80f4c059c1a99a167c7d830: Deleting WAL directory at /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-1-root/wals/082557bf92e4443cbb9d0c50a89d24c4
I20240502 20:47:09.606789 26271 catalog_manager.cc:4830] TS ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519): tablet 082557bf92e4443cbb9d0c50a89d24c4 (table test-workload [id=cc2caf6496c0475da19150577095963d]) successfully deleted
I20240502 20:47:09.780812 27232 raft_consensus.cc:1050] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec: attempting to promote NON_VOTER ae30e3bd7dc1421998cf9258aaa6bf7f to VOTER
I20240502 20:47:09.781272 27232 consensus_queue.cc:237] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 31, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } }
I20240502 20:47:09.782572 26787 raft_consensus.cc:1261] T 8ea4b2412a3a420f93d384aa48117c60 P ae30e3bd7dc1421998cf9258aaa6bf7f [term 2 LEARNER]: Refusing update from remote peer 2c2b4bb6fbe94bb8bea2eb59520f5fec: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240502 20:47:09.782609 26461 raft_consensus.cc:1261] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [term 2 FOLLOWER]: Refusing update from remote peer 2c2b4bb6fbe94bb8bea2eb59520f5fec: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240502 20:47:09.782848 26391 raft_consensus.cc:1261] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Refusing update from remote peer 2c2b4bb6fbe94bb8bea2eb59520f5fec: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240502 20:47:09.783259 27232 consensus_queue.cc:1035] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 20:47:09.783463 27191 consensus_queue.cc:1035] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 20:47:09.783638 27235 consensus_queue.cc:1035] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 20:47:09.785725 27191 raft_consensus.cc:2932] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 31 to 33, ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } }
I20240502 20:47:09.786098 26787 raft_consensus.cc:2932] T 8ea4b2412a3a420f93d384aa48117c60 P ae30e3bd7dc1421998cf9258aaa6bf7f [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 31 to 33, ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } }
I20240502 20:47:09.787750 26391 raft_consensus.cc:2932] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 31 to 33, ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } }
I20240502 20:47:09.787988 26281 catalog_manager.cc:5484] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec reported cstate change: config changed from index 31 to 33, ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:47:09.787984 26464 raft_consensus.cc:2932] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 31 to 33, ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } }
I20240502 20:47:09.791735 26787 raft_consensus.cc:1261] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f [term 2 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240502 20:47:09.792145 27216 consensus_queue.cc:1035] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 20:47:09.792227 26863 consensus_queue.cc:237] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } }
I20240502 20:47:09.793318 26786 raft_consensus.cc:1261] T 8ea4b2412a3a420f93d384aa48117c60 P ae30e3bd7dc1421998cf9258aaa6bf7f [term 2 FOLLOWER]: Refusing update from remote peer 2c2b4bb6fbe94bb8bea2eb59520f5fec: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240502 20:47:09.793812 27232 consensus_queue.cc:1035] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 20:47:09.794855 26461 raft_consensus.cc:1261] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [term 2 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240502 20:47:09.795194 27170 consensus_queue.cc:1035] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 20:47:09.796988 27232 raft_consensus.cc:2932] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 2 LEADER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } }
I20240502 20:47:09.797304 26786 raft_consensus.cc:2932] T 8ea4b2412a3a420f93d384aa48117c60 P ae30e3bd7dc1421998cf9258aaa6bf7f [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } }
I20240502 20:47:09.797598 26464 raft_consensus.cc:1261] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [term 2 FOLLOWER]: Refusing update from remote peer 2c2b4bb6fbe94bb8bea2eb59520f5fec: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240502 20:47:09.797933 27233 consensus_queue.cc:1035] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 20:47:09.798799 26330 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8ea4b2412a3a420f93d384aa48117c60 with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:47:09.799640 26283 catalog_manager.cc:5484] T 8ea4b2412a3a420f93d384aa48117c60 P 2c2b4bb6fbe94bb8bea2eb59520f5fec reported cstate change: config changed from index 33 to 34, VOTER 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129) evicted. New cstate: current_term: 2 leader_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:47:09.799664 26467 raft_consensus.cc:2932] T 8ea4b2412a3a420f93d384aa48117c60 P ddb0f390a80f4c059c1a99a167c7d830 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } }
I20240502 20:47:09.802058 26373 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ea4b2412a3a420f93d384aa48117c60 with delete_type TABLET_DATA_TOMBSTONED (TS 97406d5f0d1f4d59bdc784684baa1559 not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:52152
I20240502 20:47:09.802465 27240 tablet_replica.cc:330] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559: stopping tablet replica
I20240502 20:47:09.803146 27240 raft_consensus.cc:2229] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:09.803949 27240 raft_consensus.cc:2258] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:09.804623 27240 ts_tablet_manager.cc:1875] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:09.807039 27240 ts_tablet_manager.cc:1888] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.33
I20240502 20:47:09.807163 27240 log.cc:1196] T 8ea4b2412a3a420f93d384aa48117c60 P 97406d5f0d1f4d59bdc784684baa1559: Deleting WAL directory at /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-0-root/wals/8ea4b2412a3a420f93d384aa48117c60
I20240502 20:47:09.807333 26980 raft_consensus.cc:1050] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: attempting to promote NON_VOTER ae30e3bd7dc1421998cf9258aaa6bf7f to VOTER
I20240502 20:47:09.807718 26980 consensus_queue.cc:237] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } }
I20240502 20:47:09.808907 26273 catalog_manager.cc:4830] TS 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989): tablet 8ea4b2412a3a420f93d384aa48117c60 (table test-workload [id=cc2caf6496c0475da19150577095963d]) successfully deleted
I20240502 20:47:09.809017 26467 raft_consensus.cc:1261] T ca0df961aea849f4bc2ea340349f8848 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 20:47:09.809412 26391 raft_consensus.cc:1261] T ca0df961aea849f4bc2ea340349f8848 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 20:47:09.809584 27221 consensus_queue.cc:1035] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 20:47:09.810119 27229 consensus_queue.cc:1035] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 20:47:09.810388 26786 raft_consensus.cc:1261] T ca0df961aea849f4bc2ea340349f8848 P ae30e3bd7dc1421998cf9258aaa6bf7f [term 1 LEARNER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 20:47:09.810838 27229 consensus_queue.cc:1035] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 20:47:09.812425 27230 raft_consensus.cc:2932] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } }
I20240502 20:47:09.812909 26786 raft_consensus.cc:2932] T ca0df961aea849f4bc2ea340349f8848 P ae30e3bd7dc1421998cf9258aaa6bf7f [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } }
I20240502 20:47:09.814018 26939 raft_consensus.cc:1203] T 7d1ff41afa6e46609e6a3eb277df18ad P c8ef2de5cb854720b1090ff20c2fd13c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.31->[1.32-1.32] Dedup: 1.32->[]
I20240502 20:47:09.814685 26391 raft_consensus.cc:2932] T ca0df961aea849f4bc2ea340349f8848 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } }
I20240502 20:47:09.814978 26461 raft_consensus.cc:2932] T ca0df961aea849f4bc2ea340349f8848 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } }
I20240502 20:47:09.814683 26285 catalog_manager.cc:5484] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 reported cstate change: config changed from index 27 to 28, ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:47:09.818241 26541 consensus_queue.cc:237] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } }
I20240502 20:47:09.819283 26786 raft_consensus.cc:1261] T ca0df961aea849f4bc2ea340349f8848 P ae30e3bd7dc1421998cf9258aaa6bf7f [term 1 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 20:47:09.819506 26461 raft_consensus.cc:1261] T ca0df961aea849f4bc2ea340349f8848 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 20:47:09.819867 27200 consensus_queue.cc:1035] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 20:47:09.820377 27200 consensus_queue.cc:1035] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 20:47:09.821417 27229 raft_consensus.cc:2932] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } }
I20240502 20:47:09.821812 26786 raft_consensus.cc:2932] T ca0df961aea849f4bc2ea340349f8848 P ae30e3bd7dc1421998cf9258aaa6bf7f [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } }
I20240502 20:47:09.822057 26461 raft_consensus.cc:2932] T ca0df961aea849f4bc2ea340349f8848 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } }
I20240502 20:47:09.823505 26272 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ca0df961aea849f4bc2ea340349f8848 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:47:09.823958 26279 catalog_manager.cc:5484] T ca0df961aea849f4bc2ea340349f8848 P ae30e3bd7dc1421998cf9258aaa6bf7f reported cstate change: config changed from index 28 to 29, VOTER 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129) evicted. New cstate: current_term: 1 leader_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } }
I20240502 20:47:09.826157 26373 tablet_service.cc:1514] Processing DeleteTablet for tablet ca0df961aea849f4bc2ea340349f8848 with delete_type TABLET_DATA_TOMBSTONED (TS 97406d5f0d1f4d59bdc784684baa1559 not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:52152
I20240502 20:47:09.826382 27240 tablet_replica.cc:330] T ca0df961aea849f4bc2ea340349f8848 P 97406d5f0d1f4d59bdc784684baa1559: stopping tablet replica
I20240502 20:47:09.826530 27240 raft_consensus.cc:2229] T ca0df961aea849f4bc2ea340349f8848 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:09.826682 27240 raft_consensus.cc:2258] T ca0df961aea849f4bc2ea340349f8848 P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:09.827275 27240 ts_tablet_manager.cc:1875] T ca0df961aea849f4bc2ea340349f8848 P 97406d5f0d1f4d59bdc784684baa1559: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:09.829536 27240 ts_tablet_manager.cc:1888] T ca0df961aea849f4bc2ea340349f8848 P 97406d5f0d1f4d59bdc784684baa1559: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20240502 20:47:09.829618 27240 log.cc:1196] T ca0df961aea849f4bc2ea340349f8848 P 97406d5f0d1f4d59bdc784684baa1559: Deleting WAL directory at /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-0-root/wals/ca0df961aea849f4bc2ea340349f8848
I20240502 20:47:09.830072 26273 catalog_manager.cc:4830] TS 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989): tablet ca0df961aea849f4bc2ea340349f8848 (table test-workload [id=cc2caf6496c0475da19150577095963d]) successfully deleted
I20240502 20:47:09.965803 26391 raft_consensus.cc:1261] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 20:47:09.966240 27209 consensus_queue.cc:1035] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 20:47:09.967886 27165 raft_consensus.cc:979] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830: : Instructing follower 97406d5f0d1f4d59bdc784684baa1559 to start an election
I20240502 20:47:09.968006 27209 raft_consensus.cc:1067] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [term 2 LEADER]: Signalling peer 97406d5f0d1f4d59bdc784684baa1559 to start an election
I20240502 20:47:09.968547 26390 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7d7f28e11e1f42959251a7b03011f5dc"
dest_uuid: "97406d5f0d1f4d59bdc784684baa1559"
from {username='slave'} at 127.0.0.1:52170
I20240502 20:47:09.968672 26390 raft_consensus.cc:482] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:47:09.968726 26390 raft_consensus.cc:3037] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:47:09.969493 26538 raft_consensus.cc:1261] T 7d7f28e11e1f42959251a7b03011f5dc P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 20:47:09.969847 26390 raft_consensus.cc:504] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } }
I20240502 20:47:09.969861 27246 consensus_queue.cc:1035] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 20:47:09.970242 26390 leader_election.cc:288] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [CANDIDATE]: Term 3 election: Requested vote from peers ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519), bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881)
I20240502 20:47:09.970441 26863 raft_consensus.cc:3037] T 7d7f28e11e1f42959251a7b03011f5dc P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 1 LEARNER]: Advancing to term 2
I20240502 20:47:09.970574 26461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7f28e11e1f42959251a7b03011f5dc" candidate_uuid: "97406d5f0d1f4d59bdc784684baa1559" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830"
I20240502 20:47:09.970657 26461 raft_consensus.cc:3032] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 20:47:09.970713 26461 raft_consensus.cc:729] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ddb0f390a80f4c059c1a99a167c7d830, State: Running, Role: LEADER
I20240502 20:47:09.970825 26461 consensus_queue.cc:260] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } }
I20240502 20:47:09.970932 26461 raft_consensus.cc:3037] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:47:09.970942 26541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7f28e11e1f42959251a7b03011f5dc" candidate_uuid: "97406d5f0d1f4d59bdc784684baa1559" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3"
I20240502 20:47:09.971578 26863 raft_consensus.cc:1261] T 7d7f28e11e1f42959251a7b03011f5dc P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 2 LEARNER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 20:47:09.971757 26461 raft_consensus.cc:2454] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97406d5f0d1f4d59bdc784684baa1559 in term 3.
I20240502 20:47:09.972030 26353 leader_election.cc:302] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 97406d5f0d1f4d59bdc784684baa1559, ddb0f390a80f4c059c1a99a167c7d830; no voters: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3
I20240502 20:47:09.972224 27170 raft_consensus.cc:2781] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 20:47:09.972303 27170 raft_consensus.cc:686] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [term 3 LEADER]: Becoming Leader. State: Replica: 97406d5f0d1f4d59bdc784684baa1559, State: Running, Role: LEADER
I20240502 20:47:09.972415 27170 consensus_queue.cc:237] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } }
I20240502 20:47:09.973853 26285 catalog_manager.cc:5484] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 reported cstate change: term changed from 2 to 3, leader changed from ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130) to 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129). New cstate: current_term: 3 leader_uuid: "97406d5f0d1f4d59bdc784684baa1559" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:10.223274 27246 raft_consensus.cc:1050] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830: attempting to promote NON_VOTER c8ef2de5cb854720b1090ff20c2fd13c to VOTER
I20240502 20:47:10.223766 27246 consensus_queue.cc:237] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.32, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } }
I20240502 20:47:10.225039 26541 raft_consensus.cc:1261] T 7d1ff41afa6e46609e6a3eb277df18ad P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20240502 20:47:10.225107 26939 raft_consensus.cc:1261] T 7d1ff41afa6e46609e6a3eb277df18ad P c8ef2de5cb854720b1090ff20c2fd13c [term 1 LEARNER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20240502 20:47:10.225225 26863 raft_consensus.cc:1261] T 7d1ff41afa6e46609e6a3eb277df18ad P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20240502 20:47:10.225461 27165 consensus_queue.cc:1035] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 20:47:10.225629 27249 consensus_queue.cc:1035] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 20:47:10.225837 27209 consensus_queue.cc:1035] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 20:47:10.227694 27209 raft_consensus.cc:2932] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [term 1 LEADER]: Committing config change with OpId 1.33: config changed from index 32 to 33, c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:10.228026 26863 raft_consensus.cc:2932] T 7d1ff41afa6e46609e6a3eb277df18ad P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index 32 to 33, c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:10.228261 26939 raft_consensus.cc:2932] T 7d1ff41afa6e46609e6a3eb277df18ad P c8ef2de5cb854720b1090ff20c2fd13c [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index 32 to 33, c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:10.230131 26541 raft_consensus.cc:2932] T 7d1ff41afa6e46609e6a3eb277df18ad P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index 32 to 33, c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:10.230377 26283 catalog_manager.cc:5484] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 reported cstate change: config changed from index 32 to 33, c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ddb0f390a80f4c059c1a99a167c7d830" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:47:10.233635 26461 consensus_queue.cc:237] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 1.33, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } }
I20240502 20:47:10.234573 26939 raft_consensus.cc:1261] T 7d1ff41afa6e46609e6a3eb277df18ad P c8ef2de5cb854720b1090ff20c2fd13c [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 1 index: 34. (index mismatch)
I20240502 20:47:10.234582 26863 raft_consensus.cc:1261] T 7d1ff41afa6e46609e6a3eb277df18ad P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 1 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 1 index: 34. (index mismatch)
I20240502 20:47:10.234978 27165 consensus_queue.cc:1035] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 20:47:10.235157 27249 consensus_queue.cc:1035] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 20:47:10.236459 27249 raft_consensus.cc:2932] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 [term 1 LEADER]: Committing config change with OpId 1.34: config changed from index 33 to 34, VOTER bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:10.236876 26939 raft_consensus.cc:2932] T 7d1ff41afa6e46609e6a3eb277df18ad P c8ef2de5cb854720b1090ff20c2fd13c [term 1 FOLLOWER]: Committing config change with OpId 1.34: config changed from index 33 to 34, VOTER bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:10.238068 26863 raft_consensus.cc:2932] T 7d1ff41afa6e46609e6a3eb277df18ad P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 1 FOLLOWER]: Committing config change with OpId 1.34: config changed from index 33 to 34, VOTER bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } }
I20240502 20:47:10.238443 26271 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7d1ff41afa6e46609e6a3eb277df18ad with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:47:10.239073 26281 catalog_manager.cc:5484] T 7d1ff41afa6e46609e6a3eb277df18ad P ddb0f390a80f4c059c1a99a167c7d830 reported cstate change: config changed from index 33 to 34, VOTER bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131) evicted. New cstate: current_term: 1 leader_uuid: "ddb0f390a80f4c059c1a99a167c7d830" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:47:10.241237 26526 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d1ff41afa6e46609e6a3eb277df18ad with delete_type TABLET_DATA_TOMBSTONED (TS bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:49884
I20240502 20:47:10.241695 27253 tablet_replica.cc:330] T 7d1ff41afa6e46609e6a3eb277df18ad P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: stopping tablet replica
I20240502 20:47:10.241835 27253 raft_consensus.cc:2229] T 7d1ff41afa6e46609e6a3eb277df18ad P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:10.242136 27253 raft_consensus.cc:2258] T 7d1ff41afa6e46609e6a3eb277df18ad P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:10.242640 27253 ts_tablet_manager.cc:1875] T 7d1ff41afa6e46609e6a3eb277df18ad P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:10.244415 27253 ts_tablet_manager.cc:1888] T 7d1ff41afa6e46609e6a3eb277df18ad P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.33
I20240502 20:47:10.244482 27253 log.cc:1196] T 7d1ff41afa6e46609e6a3eb277df18ad P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Deleting WAL directory at /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-2-root/wals/7d1ff41afa6e46609e6a3eb277df18ad
I20240502 20:47:10.244871 26272 catalog_manager.cc:4830] TS bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881): tablet 7d1ff41afa6e46609e6a3eb277df18ad (table test-workload [id=cc2caf6496c0475da19150577095963d]) successfully deleted
I20240502 20:47:10.446832 27195 tablet_copy_client.cc:523] T 32631c89419245dea46abe6baa5dc295 P 2c2b4bb6fbe94bb8bea2eb59520f5fec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:47:10.448382 27195 tablet_bootstrap.cc:492] T 32631c89419245dea46abe6baa5dc295 P 2c2b4bb6fbe94bb8bea2eb59520f5fec: Bootstrap starting.
I20240502 20:47:10.467811 27195 tablet_bootstrap.cc:492] T 32631c89419245dea46abe6baa5dc295 P 2c2b4bb6fbe94bb8bea2eb59520f5fec: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=108 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:47:10.468119 27195 tablet_bootstrap.cc:492] T 32631c89419245dea46abe6baa5dc295 P 2c2b4bb6fbe94bb8bea2eb59520f5fec: Bootstrap complete.
I20240502 20:47:10.468326 27195 ts_tablet_manager.cc:1367] T 32631c89419245dea46abe6baa5dc295 P 2c2b4bb6fbe94bb8bea2eb59520f5fec: Time spent bootstrapping tablet: real 0.020s user 0.020s sys 0.000s
I20240502 20:47:10.468815 27195 raft_consensus.cc:348] T 32631c89419245dea46abe6baa5dc295 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } }
I20240502 20:47:10.468931 27195 raft_consensus.cc:729] T 32631c89419245dea46abe6baa5dc295 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 2c2b4bb6fbe94bb8bea2eb59520f5fec, State: Initialized, Role: LEARNER
I20240502 20:47:10.469029 27195 consensus_queue.cc:260] T 32631c89419245dea46abe6baa5dc295 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } }
I20240502 20:47:10.469309 27195 ts_tablet_manager.cc:1398] T 32631c89419245dea46abe6baa5dc295 P 2c2b4bb6fbe94bb8bea2eb59520f5fec: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:47:10.469913 26401 tablet_copy_service.cc:342] P 97406d5f0d1f4d59bdc784684baa1559: Request end of tablet copy session 2c2b4bb6fbe94bb8bea2eb59520f5fec-32631c89419245dea46abe6baa5dc295 received from {username='slave'} at 127.0.0.1:45116
I20240502 20:47:10.470005 26401 tablet_copy_service.cc:433] P 97406d5f0d1f4d59bdc784684baa1559: ending tablet copy session 2c2b4bb6fbe94bb8bea2eb59520f5fec-32631c89419245dea46abe6baa5dc295 on tablet 32631c89419245dea46abe6baa5dc295 with peer 2c2b4bb6fbe94bb8bea2eb59520f5fec
I20240502 20:47:10.491320 26541 raft_consensus.cc:3037] T 7d7f28e11e1f42959251a7b03011f5dc P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:47:10.492352 26541 raft_consensus.cc:1261] T 7d7f28e11e1f42959251a7b03011f5dc P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 3 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 20:47:10.492852 27255 consensus_queue.cc:1035] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 20:47:10.495271 26461 raft_consensus.cc:1261] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [term 3 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 20:47:10.495503 26863 raft_consensus.cc:3037] T 7d7f28e11e1f42959251a7b03011f5dc P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 2 LEARNER]: Advancing to term 3
I20240502 20:47:10.495704 27113 consensus_queue.cc:1035] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 20:47:10.496661 26863 raft_consensus.cc:1261] T 7d7f28e11e1f42959251a7b03011f5dc P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 3 LEARNER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 20:47:10.497212 27170 consensus_queue.cc:1035] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 20:47:10.499037 27170 raft_consensus.cc:1050] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559: attempting to promote NON_VOTER 2c2b4bb6fbe94bb8bea2eb59520f5fec to VOTER
I20240502 20:47:10.499465 27170 consensus_queue.cc:237] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 30, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } }
I20240502 20:47:10.500672 26541 raft_consensus.cc:1261] T 7d7f28e11e1f42959251a7b03011f5dc P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 3 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240502 20:47:10.500849 26863 raft_consensus.cc:1261] T 7d7f28e11e1f42959251a7b03011f5dc P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 3 LEARNER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240502 20:47:10.501039 27113 consensus_queue.cc:1035] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 20:47:10.501124 26461 raft_consensus.cc:1261] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [term 3 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240502 20:47:10.501428 27113 consensus_queue.cc:1035] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 20:47:10.501577 27255 consensus_queue.cc:1035] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 20:47:10.503109 27263 raft_consensus.cc:2932] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 29 to 32, 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:10.503494 26863 raft_consensus.cc:2932] T 7d7f28e11e1f42959251a7b03011f5dc P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 29 to 32, 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:10.503535 26461 raft_consensus.cc:2932] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 29 to 32, 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:10.505522 26279 catalog_manager.cc:5484] T 7d7f28e11e1f42959251a7b03011f5dc P 2c2b4bb6fbe94bb8bea2eb59520f5fec reported cstate change: config changed from index 29 to 32, 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "97406d5f0d1f4d59bdc784684baa1559" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:10.507877 26541 raft_consensus.cc:2932] T 7d7f28e11e1f42959251a7b03011f5dc P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 29 to 32, 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:10.509380 26390 consensus_queue.cc:237] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } }
I20240502 20:47:10.510516 26863 raft_consensus.cc:1261] T 7d7f28e11e1f42959251a7b03011f5dc P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 3 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240502 20:47:10.510845 26541 raft_consensus.cc:1261] T 7d7f28e11e1f42959251a7b03011f5dc P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 3 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240502 20:47:10.510917 27170 consensus_queue.cc:1035] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 20:47:10.511181 27255 consensus_queue.cc:1035] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 20:47:10.512436 27113 raft_consensus.cc:2932] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 [term 3 LEADER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:10.512905 26541 raft_consensus.cc:2932] T 7d7f28e11e1f42959251a7b03011f5dc P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:10.514077 26273 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7d7f28e11e1f42959251a7b03011f5dc with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:47:10.514238 26863 raft_consensus.cc:2932] T 7d7f28e11e1f42959251a7b03011f5dc P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:10.515395 26281 catalog_manager.cc:5484] T 7d7f28e11e1f42959251a7b03011f5dc P 97406d5f0d1f4d59bdc784684baa1559 reported cstate change: config changed from index 32 to 33, VOTER ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130) evicted. New cstate: current_term: 3 leader_uuid: "97406d5f0d1f4d59bdc784684baa1559" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:47:10.517593 26449 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d7f28e11e1f42959251a7b03011f5dc with delete_type TABLET_DATA_TOMBSTONED (TS ddb0f390a80f4c059c1a99a167c7d830 not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:56876
I20240502 20:47:10.517966 27264 tablet_replica.cc:330] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830: stopping tablet replica
I20240502 20:47:10.518092 27264 raft_consensus.cc:2229] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:10.518251 27264 raft_consensus.cc:2258] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:10.518784 27264 ts_tablet_manager.cc:1875] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:10.520591 27264 ts_tablet_manager.cc:1888] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.32
I20240502 20:47:10.520658 27264 log.cc:1196] T 7d7f28e11e1f42959251a7b03011f5dc P ddb0f390a80f4c059c1a99a167c7d830: Deleting WAL directory at /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-1-root/wals/7d7f28e11e1f42959251a7b03011f5dc
I20240502 20:47:10.521098 26271 catalog_manager.cc:4830] TS ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519): tablet 7d7f28e11e1f42959251a7b03011f5dc (table test-workload [id=cc2caf6496c0475da19150577095963d]) successfully deleted
I20240502 20:47:10.551472 26863 raft_consensus.cc:1203] T 32631c89419245dea46abe6baa5dc295 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 3 LEARNER]: Deduplicated request from leader. Original: 2.30->[3.31-3.31] Dedup: 3.31->[]
I20240502 20:47:11.063004 27170 raft_consensus.cc:1050] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559: attempting to promote NON_VOTER 2c2b4bb6fbe94bb8bea2eb59520f5fec to VOTER
I20240502 20:47:11.063505 27170 consensus_queue.cc:237] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 30, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } }
I20240502 20:47:11.064728 26863 raft_consensus.cc:1261] T 32631c89419245dea46abe6baa5dc295 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 3 LEARNER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240502 20:47:11.064728 26461 raft_consensus.cc:1261] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [term 3 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240502 20:47:11.064852 26541 raft_consensus.cc:1261] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 3 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240502 20:47:11.065191 27266 consensus_queue.cc:1035] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 20:47:11.065399 27268 consensus_queue.cc:1035] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 20:47:11.065572 27265 consensus_queue.cc:1035] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 20:47:11.067502 27266 raft_consensus.cc:2932] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 29 to 32, 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:11.067939 26863 raft_consensus.cc:2932] T 32631c89419245dea46abe6baa5dc295 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 29 to 32, 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:11.069319 26461 raft_consensus.cc:2932] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 29 to 32, 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:11.069630 26541 raft_consensus.cc:2932] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 29 to 32, 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:11.070066 26285 catalog_manager.cc:5484] T 32631c89419245dea46abe6baa5dc295 P 2c2b4bb6fbe94bb8bea2eb59520f5fec reported cstate change: config changed from index 29 to 32, 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "97406d5f0d1f4d59bdc784684baa1559" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:11.073779 26390 consensus_queue.cc:237] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } }
I20240502 20:47:11.074867 26863 raft_consensus.cc:1261] T 32631c89419245dea46abe6baa5dc295 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 3 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240502 20:47:11.074941 26461 raft_consensus.cc:1261] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [term 3 FOLLOWER]: Refusing update from remote peer 97406d5f0d1f4d59bdc784684baa1559: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240502 20:47:11.075253 27266 consensus_queue.cc:1035] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 20:47:11.075443 27170 consensus_queue.cc:1035] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 20:47:11.076751 27265 raft_consensus.cc:2932] T 32631c89419245dea46abe6baa5dc295 P 97406d5f0d1f4d59bdc784684baa1559 [term 3 LEADER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:11.077126 26461 raft_consensus.cc:2932] T 32631c89419245dea46abe6baa5dc295 P ddb0f390a80f4c059c1a99a167c7d830 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:11.077193 26860 raft_consensus.cc:2932] T 32631c89419245dea46abe6baa5dc295 P 2c2b4bb6fbe94bb8bea2eb59520f5fec [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:11.078143 26273 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 32631c89419245dea46abe6baa5dc295 with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:47:11.079252 26281 catalog_manager.cc:5484] T 32631c89419245dea46abe6baa5dc295 P 2c2b4bb6fbe94bb8bea2eb59520f5fec reported cstate change: config changed from index 32 to 33, VOTER bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131) evicted. New cstate: current_term: 3 leader_uuid: "97406d5f0d1f4d59bdc784684baa1559" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: false } } }
I20240502 20:47:11.081293 26526 tablet_service.cc:1514] Processing DeleteTablet for tablet 32631c89419245dea46abe6baa5dc295 with delete_type TABLET_DATA_TOMBSTONED (TS bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:49884
I20240502 20:47:11.081672 27275 tablet_replica.cc:330] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: stopping tablet replica
I20240502 20:47:11.081818 27275 raft_consensus.cc:2229] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 20:47:11.082002 27275 raft_consensus.cc:2258] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 20:47:11.082551 27275 ts_tablet_manager.cc:1875] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:47:11.085309 27275 ts_tablet_manager.cc:1888] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.32
I20240502 20:47:11.085376 27275 log.cc:1196] T 32631c89419245dea46abe6baa5dc295 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Deleting WAL directory at /tmp/dist-test-taskbspA7n/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714682726530369-22946-0/minicluster-data/ts-2-root/wals/32631c89419245dea46abe6baa5dc295
I20240502 20:47:11.085824 26272 catalog_manager.cc:4830] TS bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881): tablet 32631c89419245dea46abe6baa5dc295 (table test-workload [id=cc2caf6496c0475da19150577095963d]) successfully deleted
I20240502 20:47:11.131083 26341 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 20:47:11.132931 26541 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1bc72d9e28894957a82f3b3ef37e6f08"
dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3"
mode: GRACEFUL
new_leader_uuid: "c8ef2de5cb854720b1090ff20c2fd13c"
from {username='slave'} at 127.0.0.1:35504
I20240502 20:47:11.133102 26541 raft_consensus.cc:595] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 LEADER]: Received request to transfer leadership to TS c8ef2de5cb854720b1090ff20c2fd13c
I20240502 20:47:11.133965 26390 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5db41d5853a34dc594bafb0f4c53cd7d"
dest_uuid: "97406d5f0d1f4d59bdc784684baa1559"
mode: GRACEFUL
new_leader_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f"
from {username='slave'} at 127.0.0.1:45110
I20240502 20:47:11.134107 26390 raft_consensus.cc:595] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [term 2 LEADER]: Received request to transfer leadership to TS ae30e3bd7dc1421998cf9258aaa6bf7f
I20240502 20:47:11.134857 26390 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dfd1d70561924773aa204b5f22270b2e"
dest_uuid: "97406d5f0d1f4d59bdc784684baa1559"
mode: GRACEFUL
new_leader_uuid: "ddb0f390a80f4c059c1a99a167c7d830"
from {username='slave'} at 127.0.0.1:45110
I20240502 20:47:11.134982 26390 raft_consensus.cc:595] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559 [term 1 LEADER]: Received request to transfer leadership to TS ddb0f390a80f4c059c1a99a167c7d830
I20240502 20:47:11.135254 26341 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240502 20:47:11.135396 26341 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240502 20:47:11.241281 26335 auto_rebalancer.cc:249] could not retrieve auto-rebalancing replica moves: Not found: table cc2caf6496c0475da19150577095963d: could not find any suitable replica to move from server 97406d5f0d1f4d59bdc784684baa1559 to server c8ef2de5cb854720b1090ff20c2fd13c
I20240502 20:47:11.398000 27170 raft_consensus.cc:979] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559: : Instructing follower ae30e3bd7dc1421998cf9258aaa6bf7f to start an election
I20240502 20:47:11.398123 27170 raft_consensus.cc:1067] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [term 2 LEADER]: Signalling peer ae30e3bd7dc1421998cf9258aaa6bf7f to start an election
I20240502 20:47:11.398567 26786 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5db41d5853a34dc594bafb0f4c53cd7d"
dest_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f"
from {username='slave'} at 127.0.0.1:51670
I20240502 20:47:11.398744 26786 raft_consensus.cc:482] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:47:11.398815 26786 raft_consensus.cc:3037] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:47:11.399899 26786 raft_consensus.cc:504] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } }
I20240502 20:47:11.400303 26786 leader_election.cc:288] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f [CANDIDATE]: Term 3 election: Requested vote from peers 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989), ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519)
I20240502 20:47:11.400705 26461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5db41d5853a34dc594bafb0f4c53cd7d" candidate_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: true dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830"
I20240502 20:47:11.400740 26390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5db41d5853a34dc594bafb0f4c53cd7d" candidate_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: true dest_uuid: "97406d5f0d1f4d59bdc784684baa1559"
I20240502 20:47:11.400821 26461 raft_consensus.cc:3037] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:47:11.400904 26390 raft_consensus.cc:3032] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 20:47:11.400955 26390 raft_consensus.cc:729] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 97406d5f0d1f4d59bdc784684baa1559, State: Running, Role: LEADER
I20240502 20:47:11.401118 26390 consensus_queue.cc:260] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } }
I20240502 20:47:11.401301 26390 raft_consensus.cc:3037] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:47:11.401839 26461 raft_consensus.cc:2454] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae30e3bd7dc1421998cf9258aaa6bf7f in term 3.
I20240502 20:47:11.402150 26390 raft_consensus.cc:2454] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae30e3bd7dc1421998cf9258aaa6bf7f in term 3.
I20240502 20:47:11.402170 26750 leader_election.cc:302] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae30e3bd7dc1421998cf9258aaa6bf7f, ddb0f390a80f4c059c1a99a167c7d830; no voters:
I20240502 20:47:11.402325 27215 raft_consensus.cc:2781] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f [term 3 FOLLOWER]: Leader election won for term 3
I20240502 20:47:11.402487 27215 raft_consensus.cc:686] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f [term 3 LEADER]: Becoming Leader. State: Replica: ae30e3bd7dc1421998cf9258aaa6bf7f, State: Running, Role: LEADER
I20240502 20:47:11.402624 27215 consensus_queue.cc:237] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } }
I20240502 20:47:11.404297 26279 catalog_manager.cc:5484] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f reported cstate change: term changed from 2 to 3, leader changed from 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129) to ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132). New cstate: current_term: 3 leader_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:47:11.487118 27170 raft_consensus.cc:979] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559: : Instructing follower ddb0f390a80f4c059c1a99a167c7d830 to start an election
I20240502 20:47:11.487227 27170 raft_consensus.cc:1067] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559 [term 1 LEADER]: Signalling peer ddb0f390a80f4c059c1a99a167c7d830 to start an election
I20240502 20:47:11.487654 26461 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dfd1d70561924773aa204b5f22270b2e"
dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830"
from {username='slave'} at 127.0.0.1:56890
I20240502 20:47:11.487807 26461 raft_consensus.cc:482] T dfd1d70561924773aa204b5f22270b2e P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:47:11.487872 26461 raft_consensus.cc:3037] T dfd1d70561924773aa204b5f22270b2e P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:11.488744 26461 raft_consensus.cc:504] T dfd1d70561924773aa204b5f22270b2e P ddb0f390a80f4c059c1a99a167c7d830 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:11.489105 26461 leader_election.cc:288] T dfd1d70561924773aa204b5f22270b2e P ddb0f390a80f4c059c1a99a167c7d830 [CANDIDATE]: Term 2 election: Requested vote from peers 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129:41989), bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881)
I20240502 20:47:11.489419 26390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfd1d70561924773aa204b5f22270b2e" candidate_uuid: "ddb0f390a80f4c059c1a99a167c7d830" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "97406d5f0d1f4d59bdc784684baa1559"
I20240502 20:47:11.489419 26541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfd1d70561924773aa204b5f22270b2e" candidate_uuid: "ddb0f390a80f4c059c1a99a167c7d830" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3"
I20240502 20:47:11.489573 26541 raft_consensus.cc:3037] T dfd1d70561924773aa204b5f22270b2e P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:11.489573 26390 raft_consensus.cc:3032] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:47:11.489645 26390 raft_consensus.cc:729] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 97406d5f0d1f4d59bdc784684baa1559, State: Running, Role: LEADER
I20240502 20:47:11.489806 26390 consensus_queue.cc:260] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:11.489974 26390 raft_consensus.cc:3037] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:11.490671 26541 raft_consensus.cc:2454] T dfd1d70561924773aa204b5f22270b2e P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddb0f390a80f4c059c1a99a167c7d830 in term 2.
I20240502 20:47:11.490922 26390 raft_consensus.cc:2454] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddb0f390a80f4c059c1a99a167c7d830 in term 2.
I20240502 20:47:11.491015 26431 leader_election.cc:302] T dfd1d70561924773aa204b5f22270b2e P ddb0f390a80f4c059c1a99a167c7d830 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, ddb0f390a80f4c059c1a99a167c7d830; no voters:
I20240502 20:47:11.491284 27246 raft_consensus.cc:2781] T dfd1d70561924773aa204b5f22270b2e P ddb0f390a80f4c059c1a99a167c7d830 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:47:11.491432 27246 raft_consensus.cc:686] T dfd1d70561924773aa204b5f22270b2e P ddb0f390a80f4c059c1a99a167c7d830 [term 2 LEADER]: Becoming Leader. State: Replica: ddb0f390a80f4c059c1a99a167c7d830, State: Running, Role: LEADER
I20240502 20:47:11.491564 27246 consensus_queue.cc:237] T dfd1d70561924773aa204b5f22270b2e P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } }
I20240502 20:47:11.493044 26279 catalog_manager.cc:5484] T dfd1d70561924773aa204b5f22270b2e P ddb0f390a80f4c059c1a99a167c7d830 reported cstate change: term changed from 1 to 2, leader changed from 97406d5f0d1f4d59bdc784684baa1559 (127.22.104.129) to ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130). New cstate: current_term: 2 leader_uuid: "ddb0f390a80f4c059c1a99a167c7d830" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:11.551430 27229 raft_consensus.cc:979] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: : Instructing follower c8ef2de5cb854720b1090ff20c2fd13c to start an election
I20240502 20:47:11.551543 27229 raft_consensus.cc:1067] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 LEADER]: Signalling peer c8ef2de5cb854720b1090ff20c2fd13c to start an election
I20240502 20:47:11.551944 26939 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1bc72d9e28894957a82f3b3ef37e6f08"
dest_uuid: "c8ef2de5cb854720b1090ff20c2fd13c"
from {username='slave'} at 127.0.0.1:44676
I20240502 20:47:11.552084 26939 raft_consensus.cc:482] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:47:11.552134 26939 raft_consensus.cc:3037] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:11.553009 26939 raft_consensus.cc:504] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } }
I20240502 20:47:11.553411 26939 leader_election.cc:288] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c [CANDIDATE]: Term 2 election: Requested vote from peers ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519), bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881)
I20240502 20:47:11.553742 26541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc72d9e28894957a82f3b3ef37e6f08" candidate_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3"
I20240502 20:47:11.553761 26461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc72d9e28894957a82f3b3ef37e6f08" candidate_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "ddb0f390a80f4c059c1a99a167c7d830"
I20240502 20:47:11.553853 26541 raft_consensus.cc:3032] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:47:11.553887 26461 raft_consensus.cc:3037] T 1bc72d9e28894957a82f3b3ef37e6f08 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:11.553897 26541 raft_consensus.cc:729] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bdcd7bbd5fa14f21bbf0a7a3cdc33cb3, State: Running, Role: LEADER
I20240502 20:47:11.554073 26541 consensus_queue.cc:260] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } }
I20240502 20:47:11.554239 26541 raft_consensus.cc:3037] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:47:11.554821 26461 raft_consensus.cc:2454] T 1bc72d9e28894957a82f3b3ef37e6f08 P ddb0f390a80f4c059c1a99a167c7d830 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8ef2de5cb854720b1090ff20c2fd13c in term 2.
I20240502 20:47:11.555019 26541 raft_consensus.cc:2454] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8ef2de5cb854720b1090ff20c2fd13c in term 2.
I20240502 20:47:11.555109 26902 leader_election.cc:302] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8ef2de5cb854720b1090ff20c2fd13c, ddb0f390a80f4c059c1a99a167c7d830; no voters:
I20240502 20:47:11.555305 27227 raft_consensus.cc:2781] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:47:11.555510 27227 raft_consensus.cc:686] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c [term 2 LEADER]: Becoming Leader. State: Replica: c8ef2de5cb854720b1090ff20c2fd13c, State: Running, Role: LEADER
I20240502 20:47:11.555645 27227 consensus_queue.cc:237] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } }
I20240502 20:47:11.557193 26279 catalog_manager.cc:5484] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c reported cstate change: term changed from 1 to 2, leader changed from bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131) to c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134). New cstate: current_term: 2 leader_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:47:11.852607 26390 raft_consensus.cc:1261] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [term 3 FOLLOWER]: Refusing update from remote peer ae30e3bd7dc1421998cf9258aaa6bf7f: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240502 20:47:11.853094 27215 consensus_queue.cc:1035] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 20:47:11.855643 26461 raft_consensus.cc:1261] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [term 3 FOLLOWER]: Refusing update from remote peer ae30e3bd7dc1421998cf9258aaa6bf7f: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240502 20:47:11.856021 27215 consensus_queue.cc:1035] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 20:47:11.896903 26541 raft_consensus.cc:1261] T dfd1d70561924773aa204b5f22270b2e P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 20:47:11.897452 27246 consensus_queue.cc:1035] T dfd1d70561924773aa204b5f22270b2e P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 20:47:11.900069 26390 raft_consensus.cc:1261] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 20:47:11.900509 27246 consensus_queue.cc:1035] T dfd1d70561924773aa204b5f22270b2e P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 20:47:12.004668 26461 raft_consensus.cc:1261] T 1bc72d9e28894957a82f3b3ef37e6f08 P ddb0f390a80f4c059c1a99a167c7d830 [term 2 FOLLOWER]: Refusing update from remote peer c8ef2de5cb854720b1090ff20c2fd13c: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240502 20:47:12.005084 27227 consensus_queue.cc:1035] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 20:47:12.007412 26541 raft_consensus.cc:1261] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 FOLLOWER]: Refusing update from remote peer c8ef2de5cb854720b1090ff20c2fd13c: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240502 20:47:12.007951 27295 consensus_queue.cc:1035] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 20:47:12.244963 26541 consensus_queue.cc:237] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } }
I20240502 20:47:12.246417 26786 raft_consensus.cc:1261] T ca0df961aea849f4bc2ea340349f8848 P ae30e3bd7dc1421998cf9258aaa6bf7f [term 1 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 20:47:12.246446 26461 raft_consensus.cc:1261] T ca0df961aea849f4bc2ea340349f8848 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Refusing update from remote peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
W20240502 20:47:12.246609 26507 consensus_peers.cc:485] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 -> Peer c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134:45157): Couldn't send request to peer c8ef2de5cb854720b1090ff20c2fd13c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ca0df961aea849f4bc2ea340349f8848. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:12.246852 27229 consensus_queue.cc:1035] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 20:47:12.247037 27298 consensus_queue.cc:1035] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 20:47:12.248744 27280 raft_consensus.cc:2932] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } } }
I20240502 20:47:12.249002 26786 raft_consensus.cc:2932] T ca0df961aea849f4bc2ea340349f8848 P ae30e3bd7dc1421998cf9258aaa6bf7f [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } } }
I20240502 20:47:12.249189 26461 raft_consensus.cc:2932] T ca0df961aea849f4bc2ea340349f8848 P ddb0f390a80f4c059c1a99a167c7d830 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } } }
I20240502 20:47:12.251171 26279 catalog_manager.cc:5484] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 reported cstate change: config changed from index 29 to 30, NON_VOTER c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) added. New cstate: current_term: 1 leader_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:12.252741 26461 consensus_queue.cc:237] T dfd1d70561924773aa204b5f22270b2e P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } }
I20240502 20:47:12.254114 26390 raft_consensus.cc:1261] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 20:47:12.254614 26541 raft_consensus.cc:1261] T dfd1d70561924773aa204b5f22270b2e P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 20:47:12.254976 27246 consensus_queue.cc:1035] T dfd1d70561924773aa204b5f22270b2e P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 20:47:12.255108 27278 consensus_queue.cc:1035] T dfd1d70561924773aa204b5f22270b2e P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
W20240502 20:47:12.255587 26433 consensus_peers.cc:485] T dfd1d70561924773aa204b5f22270b2e P ddb0f390a80f4c059c1a99a167c7d830 -> Peer ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132:37635): Couldn't send request to peer ae30e3bd7dc1421998cf9258aaa6bf7f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dfd1d70561924773aa204b5f22270b2e. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:12.256669 27278 raft_consensus.cc:2932] T dfd1d70561924773aa204b5f22270b2e P ddb0f390a80f4c059c1a99a167c7d830 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } } }
I20240502 20:47:12.256986 26541 raft_consensus.cc:2932] T dfd1d70561924773aa204b5f22270b2e P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } } }
I20240502 20:47:12.259101 26283 catalog_manager.cc:5484] T dfd1d70561924773aa204b5f22270b2e P ddb0f390a80f4c059c1a99a167c7d830 reported cstate change: config changed from index -1 to 29, NON_VOTER ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) added. New cstate: current_term: 2 leader_uuid: "ddb0f390a80f4c059c1a99a167c7d830" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:12.259146 26390 raft_consensus.cc:2932] T dfd1d70561924773aa204b5f22270b2e P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } } }
I20240502 20:47:12.269284 26939 consensus_queue.cc:237] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } }
I20240502 20:47:12.270591 26541 raft_consensus.cc:1261] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 FOLLOWER]: Refusing update from remote peer c8ef2de5cb854720b1090ff20c2fd13c: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240502 20:47:12.270939 26461 raft_consensus.cc:1261] T 1bc72d9e28894957a82f3b3ef37e6f08 P ddb0f390a80f4c059c1a99a167c7d830 [term 2 FOLLOWER]: Refusing update from remote peer c8ef2de5cb854720b1090ff20c2fd13c: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240502 20:47:12.271001 27295 consensus_queue.cc:1035] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 20:47:12.271278 27295 consensus_queue.cc:1035] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 20:47:12.272475 27227 raft_consensus.cc:2932] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 31 to 33, NON_VOTER 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } } }
I20240502 20:47:12.274204 26541 raft_consensus.cc:2932] T 1bc72d9e28894957a82f3b3ef37e6f08 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 31 to 33, NON_VOTER 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } } }
I20240502 20:47:12.274192 26461 raft_consensus.cc:2932] T 1bc72d9e28894957a82f3b3ef37e6f08 P ddb0f390a80f4c059c1a99a167c7d830 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 31 to 33, NON_VOTER 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } } }
W20240502 20:47:12.274535 26902 consensus_peers.cc:485] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c -> Peer 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133:43257): Couldn't send request to peer 2c2b4bb6fbe94bb8bea2eb59520f5fec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1bc72d9e28894957a82f3b3ef37e6f08. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:12.274873 26281 catalog_manager.cc:5484] T 1bc72d9e28894957a82f3b3ef37e6f08 P c8ef2de5cb854720b1090ff20c2fd13c reported cstate change: config changed from index 31 to 33, NON_VOTER 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) added. New cstate: current_term: 2 leader_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:12.280781 26786 consensus_queue.cc:237] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 32, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } }
I20240502 20:47:12.282069 26390 raft_consensus.cc:1261] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [term 3 FOLLOWER]: Refusing update from remote peer ae30e3bd7dc1421998cf9258aaa6bf7f: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20240502 20:47:12.282406 26461 raft_consensus.cc:1261] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [term 3 FOLLOWER]: Refusing update from remote peer ae30e3bd7dc1421998cf9258aaa6bf7f: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20240502 20:47:12.282491 27285 consensus_queue.cc:1035] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 20:47:12.282743 27215 consensus_queue.cc:1035] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 20:47:12.284289 27215 raft_consensus.cc:2932] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f [term 3 LEADER]: Committing config change with OpId 3.34: config changed from index 31 to 34, NON_VOTER 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } } }
I20240502 20:47:12.284629 26390 raft_consensus.cc:2932] T 5db41d5853a34dc594bafb0f4c53cd7d P 97406d5f0d1f4d59bdc784684baa1559 [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 31 to 34, NON_VOTER 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } } }
W20240502 20:47:12.286226 26750 consensus_peers.cc:485] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f -> Peer 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133:43257): Couldn't send request to peer 2c2b4bb6fbe94bb8bea2eb59520f5fec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5db41d5853a34dc594bafb0f4c53cd7d. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:12.286598 26461 raft_consensus.cc:2932] T 5db41d5853a34dc594bafb0f4c53cd7d P ddb0f390a80f4c059c1a99a167c7d830 [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 31 to 34, NON_VOTER 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } } }
I20240502 20:47:12.286732 26283 catalog_manager.cc:5484] T 5db41d5853a34dc594bafb0f4c53cd7d P ae30e3bd7dc1421998cf9258aaa6bf7f reported cstate change: config changed from index 31 to 34, NON_VOTER 2c2b4bb6fbe94bb8bea2eb59520f5fec (127.22.104.133) added. New cstate: current_term: 3 leader_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c2b4bb6fbe94bb8bea2eb59520f5fec" member_type: NON_VOTER last_known_addr { host: "127.22.104.133" port: 43257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:12.287413 26467 consensus_queue.cc:237] T 23f60af679e74f078fa50af755a9fbf6 P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } }
I20240502 20:47:12.288759 26541 raft_consensus.cc:1261] T 23f60af679e74f078fa50af755a9fbf6 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 20:47:12.289177 27278 consensus_queue.cc:1035] T 23f60af679e74f078fa50af755a9fbf6 P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
W20240502 20:47:12.289357 26433 consensus_peers.cc:485] T 23f60af679e74f078fa50af755a9fbf6 P ddb0f390a80f4c059c1a99a167c7d830 -> Peer ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132:37635): Couldn't send request to peer ae30e3bd7dc1421998cf9258aaa6bf7f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 23f60af679e74f078fa50af755a9fbf6. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:12.290319 26390 raft_consensus.cc:1261] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Refusing update from remote peer ddb0f390a80f4c059c1a99a167c7d830: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 20:47:12.290621 27278 consensus_queue.cc:1035] T 23f60af679e74f078fa50af755a9fbf6 P ddb0f390a80f4c059c1a99a167c7d830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 20:47:12.291210 27246 raft_consensus.cc:2932] T 23f60af679e74f078fa50af755a9fbf6 P ddb0f390a80f4c059c1a99a167c7d830 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index -1 to 30, NON_VOTER ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } } }
I20240502 20:47:12.291613 26541 raft_consensus.cc:2932] T 23f60af679e74f078fa50af755a9fbf6 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index -1 to 30, NON_VOTER ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } } }
I20240502 20:47:12.293234 26279 catalog_manager.cc:5484] T 23f60af679e74f078fa50af755a9fbf6 P ddb0f390a80f4c059c1a99a167c7d830 reported cstate change: config changed from index -1 to 30, NON_VOTER ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) added. New cstate: current_term: 2 leader_uuid: "ddb0f390a80f4c059c1a99a167c7d830" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:47:12.293774 26786 consensus_queue.cc:237] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } }
I20240502 20:47:12.294651 26390 raft_consensus.cc:2932] T 23f60af679e74f078fa50af755a9fbf6 P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index -1 to 30, NON_VOTER ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } attrs { replace: true } } peers { permanent_uuid: "ddb0f390a80f4c059c1a99a167c7d830" member_type: VOTER last_known_addr { host: "127.22.104.130" port: 35519 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: NON_VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: true } } }
I20240502 20:47:12.295586 26391 raft_consensus.cc:1261] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Refusing update from remote peer ae30e3bd7dc1421998cf9258aaa6bf7f: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 20:47:12.295727 26541 raft_consensus.cc:1261] T 912fa9b5ccf74aadb519a8bf16659478 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 FOLLOWER]: Refusing update from remote peer ae30e3bd7dc1421998cf9258aaa6bf7f: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 20:47:12.296356 27215 consensus_queue.cc:1035] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 20:47:12.296983 27276 consensus_queue.cc:1035] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 20:47:12.298101 27215 raft_consensus.cc:2932] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } } }
I20240502 20:47:12.298482 26541 raft_consensus.cc:2932] T 912fa9b5ccf74aadb519a8bf16659478 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } } }
I20240502 20:47:12.300385 26390 raft_consensus.cc:2932] T 912fa9b5ccf74aadb519a8bf16659478 P 97406d5f0d1f4d59bdc784684baa1559 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } } }
I20240502 20:47:12.300585 26283 catalog_manager.cc:5484] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f reported cstate change: config changed from index 29 to 31, NON_VOTER c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134) added. New cstate: current_term: 2 leader_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "97406d5f0d1f4d59bdc784684baa1559" member_type: VOTER last_known_addr { host: "127.22.104.129" port: 41989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdcd7bbd5fa14f21bbf0a7a3cdc33cb3" member_type: VOTER last_known_addr { host: "127.22.104.131" port: 38881 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae30e3bd7dc1421998cf9258aaa6bf7f" member_type: VOTER last_known_addr { host: "127.22.104.132" port: 37635 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ef2de5cb854720b1090ff20c2fd13c" member_type: NON_VOTER last_known_addr { host: "127.22.104.134" port: 45157 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 20:47:12.301749 26752 consensus_peers.cc:485] T 912fa9b5ccf74aadb519a8bf16659478 P ae30e3bd7dc1421998cf9258aaa6bf7f -> Peer c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134:45157): Couldn't send request to peer c8ef2de5cb854720b1090ff20c2fd13c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 912fa9b5ccf74aadb519a8bf16659478. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:47:12.629864 27318 ts_tablet_manager.cc:902] T ca0df961aea849f4bc2ea340349f8848 P c8ef2de5cb854720b1090ff20c2fd13c: Initiating tablet copy from peer bdcd7bbd5fa14f21bbf0a7a3cdc33cb3 (127.22.104.131:38881)
I20240502 20:47:12.630375 27318 tablet_copy_client.cc:308] T ca0df961aea849f4bc2ea340349f8848 P c8ef2de5cb854720b1090ff20c2fd13c: tablet copy: Beginning tablet copy session from remote peer at address 127.22.104.131:38881
I20240502 20:47:12.630868 26553 tablet_copy_service.cc:140] P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Received BeginTabletCopySession request for tablet ca0df961aea849f4bc2ea340349f8848 from peer c8ef2de5cb854720b1090ff20c2fd13c ({username='slave'} at 127.0.0.1:49944)
I20240502 20:47:12.630983 26553 tablet_copy_service.cc:161] P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Beginning new tablet copy session on tablet ca0df961aea849f4bc2ea340349f8848 from peer c8ef2de5cb854720b1090ff20c2fd13c at {username='slave'} at 127.0.0.1:49944: session id = c8ef2de5cb854720b1090ff20c2fd13c-ca0df961aea849f4bc2ea340349f8848
I20240502 20:47:12.631812 26553 tablet_copy_source_session.cc:205] T ca0df961aea849f4bc2ea340349f8848 P bdcd7bbd5fa14f21bbf0a7a3cdc33cb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:47:12.632584 27318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca0df961aea849f4bc2ea340349f8848. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:47:12.634680 27318 tablet_copy_client.cc:783] T ca0df961aea849f4bc2ea340349f8848 P c8ef2de5cb854720b1090ff20c2fd13c: tablet copy: Starting download of 0 data blocks...
I20240502 20:47:12.634898 27318 tablet_copy_client.cc:647] T ca0df961aea849f4bc2ea340349f8848 P c8ef2de5cb854720b1090ff20c2fd13c: tablet copy: Starting download of 1 WAL segments...
I20240502 20:47:12.635864 27322 tablet_copy_client.cc:1002] T ca0df961aea849f4bc2ea340349f8848 P c8ef2de5cb854720b1090ff20c2fd13c: tablet copy: Injecting latency into file download: 2000
I20240502 20:47:12.636512 27323 ts_tablet_manager.cc:902] T dfd1d70561924773aa204b5f22270b2e P ae30e3bd7dc1421998cf9258aaa6bf7f: Initiating tablet copy from peer ddb0f390a80f4c059c1a99a167c7d830 (127.22.104.130:35519)
F20240502 20:47:12.636719 27323 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-082557bf92e4443cbb9d0c50a89d24c4.queue_time_us, tablet-download-pool-082557bf92e4443cbb9d0c50a89d24c4.queue_time_us) (tablet-download-pool-082557bf92e4443cbb9d0c50a89d24c4.run_cpu_time_us, tablet-download-pool-082557bf92e4443cbb9d0c50a89d24c4.run_cpu_time_us) (tablet-download-pool-082557bf92e4443cbb9d0c50a89d24c4.run_wall_time_us, tablet-download-pool-082557bf92e4443cbb9d0c50a89d24c4.run_wall_time_us) (tablet-download-pool-13f3f643f725403eb35362f87f389145.queue_time_us, tablet-download-pool-13f3f643f725403eb35362f87f389145.queue_time_us) (tablet-download-pool-13f3f643f725403eb35362f87f389145.run_cpu_time_us, tablet-download-pool-13f3f643f725403eb35362f87f389145.run_cpu_time_us) (tablet-download-pool-13f3f643f725403eb35362f87f389145.run_wall_time_us, tablet-download-pool-13f3f643f725403eb35362f87f389145.run_wall_time_us) (tablet-download-pool-1409e44c98f44bd888eea4cee4f1163a.queue_time_us, tablet-download-pool-1409e44c98f44bd888eea4cee4f1163a.queue_time_us) (tablet-download-pool-1409e44c98f44bd888eea4cee4f1163a.run_cpu_time_us, tablet-download-pool-1409e44c98f44bd888eea4cee4f1163a.run_cpu_time_us) (tablet-download-pool-1409e44c98f44bd888eea4cee4f1163a.run_wall_time_us, tablet-download-pool-1409e44c98f44bd888eea4cee4f1163a.run_wall_time_us) (tablet-download-pool-1bc72d9e28894957a82f3b3ef37e6f08.queue_time_us, tablet-download-pool-1bc72d9e28894957a82f3b3ef37e6f08.queue_time_us) (tablet-download-pool-1bc72d9e28894957a82f3b3ef37e6f08.run_cpu_time_us, tablet-download-pool-1bc72d9e28894957a82f3b3ef37e6f08.run_cpu_time_us) (tablet-download-pool-1bc72d9e28894957a82f3b3ef37e6f08.run_wall_time_us, tablet-download-pool-1bc72d9e28894957a82f3b3ef37e6f08.run_wall_time_us) (tablet-download-pool-32631c89419245dea46abe6baa5dc295.queue_time_us, tablet-download-pool-32631c89419245dea46abe6baa5dc295.queue_time_us) (tablet-download-pool-32631c89419245dea46abe6baa5dc295.run_cpu_time_us, tablet-download-pool-32631c89419245dea46abe6baa5dc295.run_cpu_time_us) (tablet-download-pool-32631c89419245dea46abe6baa5dc295.run_wall_time_us, tablet-download-pool-32631c89419245dea46abe6baa5dc295.run_wall_time_us) (tablet-download-pool-53facf0cd4384bae83d638da0d7f4e86.queue_time_us, tablet-download-pool-53facf0cd4384bae83d638da0d7f4e86.queue_time_us) (tablet-download-pool-53facf0cd4384bae83d638da0d7f4e86.run_cpu_time_us, tablet-download-pool-53facf0cd4384bae83d638da0d7f4e86.run_cpu_time_us) (tablet-download-pool-53facf0cd4384bae83d638da0d7f4e86.run_wall_time_us, tablet-download-pool-53facf0cd4384bae83d638da0d7f4e86.run_wall_time_us) (tablet-download-pool-5db41d5853a34dc594bafb0f4c53cd7d.queue_time_us, tablet-download-pool-5db41d5853a34dc594bafb0f4c53cd7d.queue_time_us) (tablet-download-pool-5db41d5853a34dc594bafb0f4c53cd7d.run_cpu_time_us, tablet-download-pool-5db41d5853a34dc594bafb0f4c53cd7d.run_cpu_time_us) (tablet-download-pool-5db41d5853a34dc594bafb0f4c53cd7d.run_wall_time_us, tablet-download-pool-5db41d5853a34dc594bafb0f4c53cd7d.run_wall_time_us) (tablet-download-pool-7d1ff41afa6e46609e6a3eb277df18ad.queue_time_us, tablet-download-pool-7d1ff41afa6e46609e6a3eb277df18ad.queue_time_us) (tablet-download-pool-7d1ff41afa6e46609e6a3eb277df18ad.run_cpu_time_us, tablet-download-pool-7d1ff41afa6e46609e6a3eb277df18ad.run_cpu_time_us) (tablet-download-pool-7d1ff41afa6e46609e6a3eb277df18ad.run_wall_time_us, tablet-download-pool-7d1ff41afa6e46609e6a3eb277df18ad.run_wall_time_us) (tablet-download-pool-7d7f28e11e1f42959251a7b03011f5dc.queue_time_us, tablet-download-pool-7d7f28e11e1f42959251a7b03011f5dc.queue_time_us) (tablet-download-pool-7d7f28e11e1f42959251a7b03011f5dc.run_cpu_time_us, tablet-download-pool-7d7f28e11e1f42959251a7b03011f5dc.run_cpu_time_us) (tablet-download-pool-7d7f28e11e1f42959251a7b03011f5dc.run_wall_time_us, tablet-download-pool-7d7f28e11e1f42959251a7b03011f5dc.run_wall_time_us) (tablet-download-pool-8ea4b2412a3a420f93d384aa48117c60.queue_time_us, tablet-download-pool-8ea4b2412a3a420f93d384aa48117c60.queue_time_us) (tablet-download-pool-8ea4b2412a3a420f93d384aa48117c60.run_cpu_time_us, tablet-download-pool-8ea4b2412a3a420f93d384aa48117c60.run_cpu_time_us) (tablet-download-pool-8ea4b2412a3a420f93d384aa48117c60.run_wall_time_us, tablet-download-pool-8ea4b2412a3a420f93d384aa48117c60.run_wall_time_us) (tablet-download-pool-912fa9b5ccf74aadb519a8bf16659478.queue_time_us, tablet-download-pool-912fa9b5ccf74aadb519a8bf16659478.queue_time_us) (tablet-download-pool-912fa9b5ccf74aadb519a8bf16659478.run_cpu_time_us, tablet-download-pool-912fa9b5ccf74aadb519a8bf16659478.run_cpu_time_us) (tablet-download-pool-912fa9b5ccf74aadb519a8bf16659478.run_wall_time_us, tablet-download-pool-912fa9b5ccf74aadb519a8bf16659478.run_wall_time_us) (tablet-download-pool-ca0df961aea849f4bc2ea340349f8848.queue_time_us, tablet-download-pool-ca0df961aea849f4bc2ea340349f8848.queue_time_us) (tablet-download-pool-ca0df961aea849f4bc2ea340349f8848.run_cpu_time_us, tablet-download-pool-ca0df961aea849f4bc2ea340349f8848.run_cpu_time_us) (tablet-download-pool-ca0df961aea849f4bc2ea340349f8848.run_wall_time_us, tablet-download-pool-ca0df961aea849f4bc2ea340349f8848.run_wall_time_us) (tablet-download-pool-dfd1d70561924773aa204b5f22270b2e.queue_time_us, tablet-download-pool-dfd1d70561924773aa204b5f22270b2e.queue_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
*** Aborted at 1714682832 (unix time) try "date -d @1714682832" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000059a2) received by PID 22946 (TID 0x7f4855ff6700) from PID 22946; stack trace: ***
@ 0x7f4913ab7040 (unknown) at ??:0
@ 0x7f4913ab6fb7 gsignal at ??:0
@ 0x7f4913ab8921 abort at ??:0
@ 0x7f49147bbdcd google::LogMessage::Fail() at ??:0
@ 0x7f49147bfb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f49147bb7cc google::LogMessage::Flush() at ??:0
@ 0x7f49147bcf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4915313044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f49152f7f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f49152f5efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f491627277a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f491627ba45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f49162fc76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f49163104f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f49162fb226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f491630ac0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240502 20:47:12.711578 27340 ts_tablet_manager.cc:902] T 1bc72d9e28894957a82f3b3ef37e6f08 P 2c2b4bb6fbe94bb8bea2eb59520f5fec: Initiating tablet copy from peer c8ef2de5cb854720b1090ff20c2fd13c (127.22.104.134:45157)
@ 0x7f49168653cc std::function<>::operator()() at ??:0
@ 0x7f49152fae06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f49152fb6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f49152fd03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f49168653cc std::function<>::operator()() at ??:0
I20240502 20:47:12.736001 27345 ts_tablet_manager.cc:902] T 912fa9b5ccf74aadb519a8bf16659478 P c8ef2de5cb854720b1090ff20c2fd13c: Initiating tablet copy from peer ae30e3bd7dc1421998cf9258aaa6bf7f (127.22.104.132:37635)
@ 0x7f49152ea807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4911aec6db start_thread at ??:0
@ 0x7f4913b9971f clone at ??:0